

<!DOCTYPE html>
<html class="writer-html5" lang="zh" >
<head>
  <meta charset="utf-8">
  
  <meta name="viewport" content="width=device-width, initial-scale=1.0">
  
  <title>选择器 &mdash; Scrapy 2.3.0 文档</title>
  

  
  <link rel="stylesheet" href="../_static/css/theme.css" type="text/css" />
  <link rel="stylesheet" href="../_static/pygments.css" type="text/css" />
  <link rel="stylesheet" href="../_static/css/tooltipster.custom.css" type="text/css" />
  <link rel="stylesheet" href="../_static/css/tooltipster.bundle.min.css" type="text/css" />
  <link rel="stylesheet" href="../_static/css/tooltipster-sideTip-shadow.min.css" type="text/css" />
  <link rel="stylesheet" href="../_static/css/tooltipster-sideTip-punk.min.css" type="text/css" />
  <link rel="stylesheet" href="../_static/css/tooltipster-sideTip-noir.min.css" type="text/css" />
  <link rel="stylesheet" href="../_static/css/tooltipster-sideTip-light.min.css" type="text/css" />
  <link rel="stylesheet" href="../_static/css/tooltipster-sideTip-borderless.min.css" type="text/css" />
  <link rel="stylesheet" href="../_static/css/micromodal.css" type="text/css" />
  <link rel="stylesheet" href="../_static/css/sphinx_rtd_theme.css" type="text/css" />

  
  
  
  

  
  <!--[if lt IE 9]>
    <script src="../_static/js/html5shiv.min.js"></script>
  <![endif]-->
  
    
      <script type="text/javascript" id="documentation_options" data-url_root="../" src="../_static/documentation_options.js"></script>
        <script src="../_static/jquery.js"></script>
        <script src="../_static/underscore.js"></script>
        <script src="../_static/doctools.js"></script>
        <script src="../_static/language_data.js"></script>
        <script src="../_static/js/hoverxref.js"></script>
        <script src="../_static/js/tooltipster.bundle.min.js"></script>
        <script src="../_static/js/micromodal.min.js"></script>
    
    <script type="text/javascript" src="../_static/js/theme.js"></script>

    
    <link rel="index" title="索引" href="../genindex.html" />
    <link rel="search" title="搜索" href="../search.html" />
    <link rel="next" title="项目" href="items.html" />
    <link rel="prev" title="蜘蛛" href="spiders.html" /> 
</head>

<body class="wy-body-for-nav">

   
  <div class="wy-grid-for-nav">
    
    <nav data-toggle="wy-nav-shift" class="wy-nav-side">
      <div class="wy-side-scroll">
        <div class="wy-side-nav-search" >
          

          
            <a href="../index.html" class="icon icon-home" alt="Documentation Home"> Scrapy
          

          
          </a>

          
            
            
              <div class="version">
                2.3
              </div>
            
          

          
<div role="search">
  <form id="rtd-search-form" class="wy-form" action="../search.html" method="get">
    <input type="text" name="q" placeholder="Search docs" />
    <input type="hidden" name="check_keywords" value="yes" />
    <input type="hidden" name="area" value="default" />
  </form>
</div>

          
        </div>

        
        <div class="wy-menu wy-menu-vertical" data-spy="affix" role="navigation" aria-label="main navigation">
          
            
            
              
            
            
              <p class="caption"><span class="caption-text">第一步</span></p>
<ul>
<li class="toctree-l1"><a class="reference internal" href="../intro/overview.html">Scrapy一目了然</a></li>
<li class="toctree-l1"><a class="reference internal" href="../intro/install.html">安装指南</a></li>
<li class="toctree-l1"><a class="reference internal" href="../intro/tutorial.html">Scrapy 教程</a></li>
<li class="toctree-l1"><a class="reference internal" href="../intro/examples.html">实例</a></li>
</ul>
<p class="caption"><span class="caption-text">基本概念</span></p>
<ul class="current">
<li class="toctree-l1"><a class="reference internal" href="commands.html">命令行工具</a></li>
<li class="toctree-l1"><a class="reference internal" href="spiders.html">蜘蛛</a></li>
<li class="toctree-l1 current"><a class="current reference internal" href="#">选择器</a><ul>
<li class="toctree-l2"><a class="reference internal" href="#using-selectors">使用选择器</a><ul>
<li class="toctree-l3"><a class="reference internal" href="#constructing-selectors">构造选择器</a></li>
<li class="toctree-l3"><a class="reference internal" href="#id1">使用选择器</a></li>
<li class="toctree-l3"><a class="reference internal" href="#extensions-to-css-selectors">CSS选择器的扩展</a></li>
<li class="toctree-l3"><a class="reference internal" href="#nesting-selectors">嵌套选择器</a></li>
<li class="toctree-l3"><a class="reference internal" href="#selecting-element-attributes">选择元素属性</a></li>
<li class="toctree-l3"><a class="reference internal" href="#using-selectors-with-regular-expressions">将选择器与正则表达式一起使用</a></li>
<li class="toctree-l3"><a class="reference internal" href="#extract-and-extract-first">extract（）和extract_first（）。</a></li>
</ul>
</li>
<li class="toctree-l2"><a class="reference internal" href="#working-with-xpaths">使用xpaths</a><ul>
<li class="toctree-l3"><a class="reference internal" href="#working-with-relative-xpaths">使用相对路径</a></li>
<li class="toctree-l3"><a class="reference internal" href="#when-querying-by-class-consider-using-css">按类查询时，请考虑使用CSS</a></li>
<li class="toctree-l3"><a class="reference internal" href="#beware-of-the-difference-between-node-1-and-node-1">注意//节点之间的区别 [1] 和（/ /节点） [1]</a></li>
<li class="toctree-l3"><a class="reference internal" href="#using-text-nodes-in-a-condition">在条件中使用文本节点</a></li>
<li class="toctree-l3"><a class="reference internal" href="#variables-in-xpath-expressions">xpath表达式中的变量</a></li>
<li class="toctree-l3"><a class="reference internal" href="#removing-namespaces">正在删除命名空间</a></li>
<li class="toctree-l3"><a class="reference internal" href="#using-exslt-extensions">使用exslt扩展</a><ul>
<li class="toctree-l4"><a class="reference internal" href="#regular-expressions">正则表达式</a></li>
<li class="toctree-l4"><a class="reference internal" href="#set-operations">集合运算</a></li>
</ul>
</li>
<li class="toctree-l3"><a class="reference internal" href="#other-xpath-extensions">其他XPath扩展</a></li>
</ul>
</li>
<li class="toctree-l2"><a class="reference internal" href="#module-scrapy.selector">内置选择器引用</a><ul>
<li class="toctree-l3"><a class="reference internal" href="#selector-objects">选择器对象</a></li>
<li class="toctree-l3"><a class="reference internal" href="#selectorlist-objects">SelectorList对象</a></li>
</ul>
</li>
<li class="toctree-l2"><a class="reference internal" href="#examples">实例</a><ul>
<li class="toctree-l3"><a class="reference internal" href="#selector-examples-on-html-response">HTML响应的选择器示例</a></li>
<li class="toctree-l3"><a class="reference internal" href="#selector-examples-on-xml-response">XML响应的选择器示例</a></li>
</ul>
</li>
</ul>
</li>
<li class="toctree-l1"><a class="reference internal" href="items.html">项目</a></li>
<li class="toctree-l1"><a class="reference internal" href="loaders.html">项目加载器</a></li>
<li class="toctree-l1"><a class="reference internal" href="shell.html">Scrapy shell</a></li>
<li class="toctree-l1"><a class="reference internal" href="item-pipeline.html">项目管道</a></li>
<li class="toctree-l1"><a class="reference internal" href="feed-exports.html">Feed 导出</a></li>
<li class="toctree-l1"><a class="reference internal" href="request-response.html">请求和响应</a></li>
<li class="toctree-l1"><a class="reference internal" href="link-extractors.html">链接提取器</a></li>
<li class="toctree-l1"><a class="reference internal" href="settings.html">设置</a></li>
<li class="toctree-l1"><a class="reference internal" href="exceptions.html">例外情况</a></li>
</ul>
<p class="caption"><span class="caption-text">内置服务</span></p>
<ul>
<li class="toctree-l1"><a class="reference internal" href="logging.html">登录</a></li>
<li class="toctree-l1"><a class="reference internal" href="stats.html">统计数据集合</a></li>
<li class="toctree-l1"><a class="reference internal" href="email.html">发送电子邮件</a></li>
<li class="toctree-l1"><a class="reference internal" href="telnetconsole.html">远程登录控制台</a></li>
<li class="toctree-l1"><a class="reference internal" href="webservice.html">Web服务</a></li>
</ul>
<p class="caption"><span class="caption-text">解决具体问题</span></p>
<ul>
<li class="toctree-l1"><a class="reference internal" href="../faq.html">常见问题</a></li>
<li class="toctree-l1"><a class="reference internal" href="debug.html">调试spiders</a></li>
<li class="toctree-l1"><a class="reference internal" href="contracts.html">蜘蛛合约</a></li>
<li class="toctree-l1"><a class="reference internal" href="practices.html">常用做法</a></li>
<li class="toctree-l1"><a class="reference internal" href="broad-crawls.html">宽爬行</a></li>
<li class="toctree-l1"><a class="reference internal" href="developer-tools.html">使用浏览器的开发人员工具进行抓取</a></li>
<li class="toctree-l1"><a class="reference internal" href="dynamic-content.html">选择动态加载的内容</a></li>
<li class="toctree-l1"><a class="reference internal" href="leaks.html">调试内存泄漏</a></li>
<li class="toctree-l1"><a class="reference internal" href="media-pipeline.html">下载和处理文件和图像</a></li>
<li class="toctree-l1"><a class="reference internal" href="deploy.html">部署蜘蛛</a></li>
<li class="toctree-l1"><a class="reference internal" href="autothrottle.html">AutoThrottle 扩展</a></li>
<li class="toctree-l1"><a class="reference internal" href="benchmarking.html">标杆管理</a></li>
<li class="toctree-l1"><a class="reference internal" href="jobs.html">作业：暂停和恢复爬行</a></li>
<li class="toctree-l1"><a class="reference internal" href="coroutines.html">协同程序</a></li>
<li class="toctree-l1"><a class="reference internal" href="asyncio.html">asyncio</a></li>
</ul>
<p class="caption"><span class="caption-text">扩展Scrapy</span></p>
<ul>
<li class="toctree-l1"><a class="reference internal" href="architecture.html">体系结构概述</a></li>
<li class="toctree-l1"><a class="reference internal" href="downloader-middleware.html">下载器中间件</a></li>
<li class="toctree-l1"><a class="reference internal" href="spider-middleware.html">蜘蛛中间件</a></li>
<li class="toctree-l1"><a class="reference internal" href="extensions.html">扩展</a></li>
<li class="toctree-l1"><a class="reference internal" href="api.html">核心API</a></li>
<li class="toctree-l1"><a class="reference internal" href="signals.html">信号</a></li>
<li class="toctree-l1"><a class="reference internal" href="exporters.html">条目导出器</a></li>
</ul>
<p class="caption"><span class="caption-text">其余所有</span></p>
<ul>
<li class="toctree-l1"><a class="reference internal" href="../news.html">发行说明</a></li>
<li class="toctree-l1"><a class="reference internal" href="../contributing.html">为 Scrapy 贡献</a></li>
<li class="toctree-l1"><a class="reference internal" href="../versioning.html">版本控制和API稳定性</a></li>
</ul>

            
          
        </div>
        
      </div>
    </nav>

    <section data-toggle="wy-nav-shift" class="wy-nav-content-wrap">

      
      <nav class="wy-nav-top" aria-label="top navigation">
        
          <i data-toggle="wy-nav-top" class="fa fa-bars"></i>
          <a href="../index.html">Scrapy</a>
        
      </nav>


      <div class="wy-nav-content">
        
        <div class="rst-content">
        
          















<div role="navigation" aria-label="breadcrumbs navigation">

  <ul class="wy-breadcrumbs">
    
      <li><a href="../index.html" class="icon icon-home"></a> &raquo;</li>
        
      <li>选择器</li>
    
    
      <li class="wy-breadcrumbs-aside">
        
            
        
      </li>
    
  </ul>

  
  <hr/>
</div>
          <div role="main" class="document" itemscope="itemscope" itemtype="http://schema.org/Article">
           <div itemprop="articleBody">
            
  <div class="section" id="selectors">
<span id="topics-selectors"></span><h1>选择器<a class="headerlink" href="#selectors" title="永久链接至标题">¶</a></h1>
<p>当你抓取网页时，你需要执行的最常见的任务是从HTML源代码中提取数据。有几个库可以实现这一点，例如：</p>
<blockquote>
<div><ul class="simple">
<li><p><a class="reference external" href="https://www.crummy.com/software/BeautifulSoup/">BeautifulSoup</a> 在Python程序员中是一个非常流行的Web抓取库，它基于HTML代码的结构构造了一个Python对象，并且能够很好地处理错误的标记，但是它有一个缺点：速度慢。</p><script async src="https://pagead2.googlesyndication.com/pagead/js/adsbygoogle.js"></script>
<ins class="adsbygoogle"
     style="display:block; text-align:center;"
     data-ad-layout="in-article"
     data-ad-format="fluid"
     data-ad-client="ca-pub-1466963416408457"
     data-ad-slot="8850786025"></ins>
<script>
     (adsbygoogle = window.adsbygoogle || []).push({});
</script></li>
<li><p><a class="reference external" href="https://lxml.de/">lxml</a> 是一个XML解析库（它也解析HTML），使用基于 <a class="reference external" href="https://docs.python.org/3/library/xml.etree.elementtree.html#module-xml.etree.ElementTree" title="(在 Python v3.9)"><code class="xref py py-mod docutils literal notranslate"><span class="pre">ElementTree</span></code></a> . （LXML不是Python标准库的一部分。）</p></li>
</ul>
</div></blockquote>
<p>Scrapy有自己的数据提取机制。它们被称为选择器，因为它们“选择”HTML文档的某些部分 <a class="reference external" href="https://www.w3.org/TR/xpath/all/">XPath</a> 或 <a class="reference external" href="https://www.w3.org/TR/selectors">CSS</a> 表达。</p>
<p><a class="reference external" href="https://www.w3.org/TR/xpath/all/">XPath</a> 是一种在XML文档中选择节点的语言，也可以与HTML一起使用。 <a class="reference external" href="https://www.w3.org/TR/selectors">CSS</a> 是用于将样式应用于HTML文档的语言。它定义选择器，将这些样式与特定的HTML元素相关联。</p>
<div class="admonition note">
<p class="admonition-title">注解</p>
<p>碎屑选择器是一个很薄的包装 <a class="reference external" href="https://parsel.readthedocs.io/en/latest/">parsel</a> library；这个包装器的目的是提供更好的与slapy响应对象的集成。</p>
<p><a class="reference external" href="https://parsel.readthedocs.io/en/latest/">parsel</a> 是一个独立的网页爬取库，可以使用没有 Scrapy  。它使用 <a class="reference external" href="https://lxml.de/">lxml</a> 库位于引擎盖下，并在LXML API之上实现一个简单的API。这意味着scrapy选择器在速度和解析精度方面与lxml非常相似。</p>
</div>
<div class="section" id="using-selectors">
<h2>使用选择器<a class="headerlink" href="#using-selectors" title="永久链接至标题">¶</a></h2>
<div class="section" id="constructing-selectors">
<h3>构造选择器<a class="headerlink" href="#constructing-selectors" title="永久链接至标题">¶</a></h3>
<p>响应对象公开 <a class="reference internal" href="#scrapy.selector.Selector" title="scrapy.selector.Selector"><code class="xref py py-class docutils literal notranslate"><span class="pre">Selector</span></code></a> 实例对 <code class="docutils literal notranslate"><span class="pre">.selector</span></code> 属性：</p>
<div class="doctest highlight-default notranslate"><div class="highlight"><pre><span></span><span class="gp">&gt;&gt;&gt; </span><span class="n">response</span><span class="o">.</span><span class="n">selector</span><span class="o">.</span><span class="n">xpath</span><span class="p">(</span><span class="s1">&#39;//span/text()&#39;</span><span class="p">)</span><span class="o">.</span><span class="n">get</span><span class="p">()</span>
<span class="go">&#39;good&#39;</span>
</pre></div>
</div>
<p>使用xpath和css查询响应非常常见，因此响应中还包含两个快捷方式： <code class="docutils literal notranslate"><span class="pre">response.xpath()</span></code> 和 <code class="docutils literal notranslate"><span class="pre">response.css()</span></code> ：</p>
<div class="doctest highlight-default notranslate"><div class="highlight"><pre><span></span><span class="gp">&gt;&gt;&gt; </span><span class="n">response</span><span class="o">.</span><span class="n">xpath</span><span class="p">(</span><span class="s1">&#39;//span/text()&#39;</span><span class="p">)</span><span class="o">.</span><span class="n">get</span><span class="p">()</span>
<span class="go">&#39;good&#39;</span>
<span class="gp">&gt;&gt;&gt; </span><span class="n">response</span><span class="o">.</span><span class="n">css</span><span class="p">(</span><span class="s1">&#39;span::text&#39;</span><span class="p">)</span><span class="o">.</span><span class="n">get</span><span class="p">()</span>
<span class="go">&#39;good&#39;</span>
</pre></div>
</div>
<p>Scrapy选择器是 <a class="reference internal" href="#scrapy.selector.Selector" title="scrapy.selector.Selector"><code class="xref py py-class docutils literal notranslate"><span class="pre">Selector</span></code></a> 通过传递 <a class="reference internal" href="request-response.html#scrapy.http.TextResponse" title="scrapy.http.TextResponse"><code class="xref py py-class docutils literal notranslate"><span class="pre">TextResponse</span></code></a> 对象或标记作为字符串（在 <code class="docutils literal notranslate"><span class="pre">text</span></code> 争论）。</p>
<p>通常不需要手动构造垃圾选择器： <code class="docutils literal notranslate"><span class="pre">response</span></code> 对象在spider回调中可用，因此在大多数情况下使用它更方便 <code class="docutils literal notranslate"><span class="pre">response.css()</span></code> 和 <code class="docutils literal notranslate"><span class="pre">response.xpath()</span></code> 捷径。通过使用 <code class="docutils literal notranslate"><span class="pre">response.selector</span></code> 或者这些快捷方式之一，您还可以确保响应主体只解析一次。</p>
<p>但如果需要，可以使用 <code class="docutils literal notranslate"><span class="pre">Selector</span></code> 直接。从文本构建：</p>
<div class="doctest highlight-default notranslate"><div class="highlight"><pre><span></span><span class="gp">&gt;&gt;&gt; </span><span class="kn">from</span> <span class="nn">scrapy.selector</span> <span class="kn">import</span> <span class="n">Selector</span>
<span class="gp">&gt;&gt;&gt; </span><span class="n">body</span> <span class="o">=</span> <span class="s1">&#39;&lt;html&gt;&lt;body&gt;&lt;span&gt;good&lt;/span&gt;&lt;/body&gt;&lt;/html&gt;&#39;</span>
<span class="gp">&gt;&gt;&gt; </span><span class="n">Selector</span><span class="p">(</span><span class="n">text</span><span class="o">=</span><span class="n">body</span><span class="p">)</span><span class="o">.</span><span class="n">xpath</span><span class="p">(</span><span class="s1">&#39;//span/text()&#39;</span><span class="p">)</span><span class="o">.</span><span class="n">get</span><span class="p">()</span>
<span class="go">&#39;good&#39;</span>
</pre></div>
</div>
<p>从响应构造- <a class="reference internal" href="request-response.html#scrapy.http.HtmlResponse" title="scrapy.http.HtmlResponse"><code class="xref py py-class docutils literal notranslate"><span class="pre">HtmlResponse</span></code></a> 是其中之一 <a class="reference internal" href="request-response.html#scrapy.http.TextResponse" title="scrapy.http.TextResponse"><code class="xref py py-class docutils literal notranslate"><span class="pre">TextResponse</span></code></a> 子类：</p>
<div class="doctest highlight-default notranslate"><div class="highlight"><pre><span></span><span class="gp">&gt;&gt;&gt; </span><span class="kn">from</span> <span class="nn">scrapy.selector</span> <span class="kn">import</span> <span class="n">Selector</span>
<span class="gp">&gt;&gt;&gt; </span><span class="kn">from</span> <span class="nn">scrapy.http</span> <span class="kn">import</span> <span class="n">HtmlResponse</span>
<span class="gp">&gt;&gt;&gt; </span><span class="n">response</span> <span class="o">=</span> <span class="n">HtmlResponse</span><span class="p">(</span><span class="n">url</span><span class="o">=</span><span class="s1">&#39;http://example.com&#39;</span><span class="p">,</span> <span class="n">body</span><span class="o">=</span><span class="n">body</span><span class="p">)</span>
<span class="gp">&gt;&gt;&gt; </span><span class="n">Selector</span><span class="p">(</span><span class="n">response</span><span class="o">=</span><span class="n">response</span><span class="p">)</span><span class="o">.</span><span class="n">xpath</span><span class="p">(</span><span class="s1">&#39;//span/text()&#39;</span><span class="p">)</span><span class="o">.</span><span class="n">get</span><span class="p">()</span>
<span class="go">&#39;good&#39;</span>
</pre></div>
</div>
<p><code class="docutils literal notranslate"><span class="pre">Selector</span></code> 根据输入类型自动选择最佳的解析规则（XML对HTML）。</p>
</div>
<div class="section" id="id1">
<h3>使用选择器<a class="headerlink" href="#id1" title="永久链接至标题">¶</a></h3>
<p>为了解释如何使用选择器，我们将使用 <code class="docutils literal notranslate"><span class="pre">Scrapy</span> <span class="pre">shell</span></code> （提供交互式测试）和位于Scrapy文档服务器中的示例页面：</p>
<blockquote>
<div><p><a class="reference external" href="https://docs.scrapy.org/en/latest/_static/selectors-sample1.html">https://docs.scrapy.org/en/latest/_static/selectors-sample1.html</a></p>
</div></blockquote>
<p id="topics-selectors-htmlcode">为了完整起见，下面是完整的HTML代码：</p>
<div class="highlight-html notranslate"><div class="highlight"><pre><span></span><span class="p">&lt;</span><span class="nt">html</span><span class="p">&gt;</span>
 <span class="p">&lt;</span><span class="nt">head</span><span class="p">&gt;</span>
  <span class="p">&lt;</span><span class="nt">base</span> <span class="na">href</span><span class="o">=</span><span class="s">&#39;http://example.com/&#39;</span> <span class="p">/&gt;</span>
  <span class="p">&lt;</span><span class="nt">title</span><span class="p">&gt;</span>Example website<span class="p">&lt;/</span><span class="nt">title</span><span class="p">&gt;</span>
 <span class="p">&lt;/</span><span class="nt">head</span><span class="p">&gt;</span>
 <span class="p">&lt;</span><span class="nt">body</span><span class="p">&gt;</span>
  <span class="p">&lt;</span><span class="nt">div</span> <span class="na">id</span><span class="o">=</span><span class="s">&#39;images&#39;</span><span class="p">&gt;</span>
   <span class="p">&lt;</span><span class="nt">a</span> <span class="na">href</span><span class="o">=</span><span class="s">&#39;image1.html&#39;</span><span class="p">&gt;</span>Name: My image 1 <span class="p">&lt;</span><span class="nt">br</span> <span class="p">/&gt;&lt;</span><span class="nt">img</span> <span class="na">src</span><span class="o">=</span><span class="s">&#39;image1_thumb.jpg&#39;</span> <span class="p">/&gt;&lt;/</span><span class="nt">a</span><span class="p">&gt;</span>
   <span class="p">&lt;</span><span class="nt">a</span> <span class="na">href</span><span class="o">=</span><span class="s">&#39;image2.html&#39;</span><span class="p">&gt;</span>Name: My image 2 <span class="p">&lt;</span><span class="nt">br</span> <span class="p">/&gt;&lt;</span><span class="nt">img</span> <span class="na">src</span><span class="o">=</span><span class="s">&#39;image2_thumb.jpg&#39;</span> <span class="p">/&gt;&lt;/</span><span class="nt">a</span><span class="p">&gt;</span>
   <span class="p">&lt;</span><span class="nt">a</span> <span class="na">href</span><span class="o">=</span><span class="s">&#39;image3.html&#39;</span><span class="p">&gt;</span>Name: My image 3 <span class="p">&lt;</span><span class="nt">br</span> <span class="p">/&gt;&lt;</span><span class="nt">img</span> <span class="na">src</span><span class="o">=</span><span class="s">&#39;image3_thumb.jpg&#39;</span> <span class="p">/&gt;&lt;/</span><span class="nt">a</span><span class="p">&gt;</span>
   <span class="p">&lt;</span><span class="nt">a</span> <span class="na">href</span><span class="o">=</span><span class="s">&#39;image4.html&#39;</span><span class="p">&gt;</span>Name: My image 4 <span class="p">&lt;</span><span class="nt">br</span> <span class="p">/&gt;&lt;</span><span class="nt">img</span> <span class="na">src</span><span class="o">=</span><span class="s">&#39;image4_thumb.jpg&#39;</span> <span class="p">/&gt;&lt;/</span><span class="nt">a</span><span class="p">&gt;</span>
   <span class="p">&lt;</span><span class="nt">a</span> <span class="na">href</span><span class="o">=</span><span class="s">&#39;image5.html&#39;</span><span class="p">&gt;</span>Name: My image 5 <span class="p">&lt;</span><span class="nt">br</span> <span class="p">/&gt;&lt;</span><span class="nt">img</span> <span class="na">src</span><span class="o">=</span><span class="s">&#39;image5_thumb.jpg&#39;</span> <span class="p">/&gt;&lt;/</span><span class="nt">a</span><span class="p">&gt;</span>
  <span class="p">&lt;/</span><span class="nt">div</span><span class="p">&gt;</span>
 <span class="p">&lt;/</span><span class="nt">body</span><span class="p">&gt;</span>
<span class="p">&lt;/</span><span class="nt">html</span><span class="p">&gt;</span>
</pre></div>
</div>
<p>首先，让我们打开外壳：</p>
<div class="highlight-sh notranslate"><div class="highlight"><pre><span></span>scrapy shell https://docs.scrapy.org/en/latest/_static/selectors-sample1.html
</pre></div>
</div>
<p>然后，在shell加载之后，您将得到可用的响应 <code class="docutils literal notranslate"><span class="pre">response</span></code> shell变量及其附加的选择器 <code class="docutils literal notranslate"><span class="pre">response.selector</span></code> 属性。</p>
<p>由于我们处理的是HTML，选择器将自动使用HTML解析器。</p>
<p>所以，通过观察 <a class="reference internal" href="#topics-selectors-htmlcode"><span class="std std-ref">HTML code</span></a> 在该页面中，让我们构造一个XPath来选择title标记内的文本：</p>
<div class="doctest highlight-default notranslate"><div class="highlight"><pre><span></span><span class="gp">&gt;&gt;&gt; </span><span class="n">response</span><span class="o">.</span><span class="n">xpath</span><span class="p">(</span><span class="s1">&#39;//title/text()&#39;</span><span class="p">)</span>
<span class="go">[&lt;Selector xpath=&#39;//title/text()&#39; data=&#39;Example website&#39;&gt;]</span>
</pre></div>
</div>
<p>要实际提取文本数据，必须调用选择器 <code class="docutils literal notranslate"><span class="pre">.get()</span></code> 或 <code class="docutils literal notranslate"><span class="pre">.getall()</span></code> 方法如下：</p>
<div class="doctest highlight-default notranslate"><div class="highlight"><pre><span></span><span class="gp">&gt;&gt;&gt; </span><span class="n">response</span><span class="o">.</span><span class="n">xpath</span><span class="p">(</span><span class="s1">&#39;//title/text()&#39;</span><span class="p">)</span><span class="o">.</span><span class="n">getall</span><span class="p">()</span>
<span class="go">[&#39;Example website&#39;]</span>
<span class="gp">&gt;&gt;&gt; </span><span class="n">response</span><span class="o">.</span><span class="n">xpath</span><span class="p">(</span><span class="s1">&#39;//title/text()&#39;</span><span class="p">)</span><span class="o">.</span><span class="n">get</span><span class="p">()</span>
<span class="go">&#39;Example website&#39;</span>
</pre></div>
</div>
<p><code class="docutils literal notranslate"><span class="pre">.get()</span></code> 始终返回单个结果；如果有多个匹配项，则返回第一个匹配项的内容；如果没有匹配项，则不返回任何匹配项。 <code class="docutils literal notranslate"><span class="pre">.getall()</span></code> 返回包含所有结果的列表。</p>
<p>请注意，CSS选择器可以使用CSS3伪元素选择文本或属性节点：</p>
<div class="doctest highlight-default notranslate"><div class="highlight"><pre><span></span><span class="gp">&gt;&gt;&gt; </span><span class="n">response</span><span class="o">.</span><span class="n">css</span><span class="p">(</span><span class="s1">&#39;title::text&#39;</span><span class="p">)</span><span class="o">.</span><span class="n">get</span><span class="p">()</span>
<span class="go">&#39;Example website&#39;</span>
</pre></div>
</div>
<p>正如你所看到的， <code class="docutils literal notranslate"><span class="pre">.xpath()</span></code> 和 <code class="docutils literal notranslate"><span class="pre">.css()</span></code> 方法返回 <a class="reference internal" href="#scrapy.selector.SelectorList" title="scrapy.selector.SelectorList"><code class="xref py py-class docutils literal notranslate"><span class="pre">SelectorList</span></code></a> 实例，它是新选择器的列表。此API可用于快速选择嵌套数据：</p>
<div class="doctest highlight-default notranslate"><div class="highlight"><pre><span></span><span class="gp">&gt;&gt;&gt; </span><span class="n">response</span><span class="o">.</span><span class="n">css</span><span class="p">(</span><span class="s1">&#39;img&#39;</span><span class="p">)</span><span class="o">.</span><span class="n">xpath</span><span class="p">(</span><span class="s1">&#39;@src&#39;</span><span class="p">)</span><span class="o">.</span><span class="n">getall</span><span class="p">()</span>
<span class="go">[&#39;image1_thumb.jpg&#39;,</span>
<span class="go"> &#39;image2_thumb.jpg&#39;,</span>
<span class="go"> &#39;image3_thumb.jpg&#39;,</span>
<span class="go"> &#39;image4_thumb.jpg&#39;,</span>
<span class="go"> &#39;image5_thumb.jpg&#39;]</span>
</pre></div>
</div>
<p>如果只提取第一个匹配的元素，则可以调用选择器 <code class="docutils literal notranslate"><span class="pre">.get()</span></code> （或其别名） <code class="docutils literal notranslate"><span class="pre">.extract_first()</span></code> 常用于旧版本）：</p>
<div class="doctest highlight-default notranslate"><div class="highlight"><pre><span></span><span class="gp">&gt;&gt;&gt; </span><span class="n">response</span><span class="o">.</span><span class="n">xpath</span><span class="p">(</span><span class="s1">&#39;//div[@id=&quot;images&quot;]/a/text()&#39;</span><span class="p">)</span><span class="o">.</span><span class="n">get</span><span class="p">()</span>
<span class="go">&#39;Name: My image 1 &#39;</span>
</pre></div>
</div>
<p>它返回 <code class="docutils literal notranslate"><span class="pre">None</span></code> 如果找不到元素：</p>
<div class="doctest highlight-default notranslate"><div class="highlight"><pre><span></span><span class="gp">&gt;&gt;&gt; </span><span class="n">response</span><span class="o">.</span><span class="n">xpath</span><span class="p">(</span><span class="s1">&#39;//div[@id=&quot;not-exists&quot;]/text()&#39;</span><span class="p">)</span><span class="o">.</span><span class="n">get</span><span class="p">()</span> <span class="ow">is</span> <span class="kc">None</span>
<span class="go">True</span>
</pre></div>
</div>
<p>可以将默认返回值作为参数提供，以代替 <code class="docutils literal notranslate"><span class="pre">None</span></code> ：</p>
<div class="doctest highlight-default notranslate"><div class="highlight"><pre><span></span><span class="gp">&gt;&gt;&gt; </span><span class="n">response</span><span class="o">.</span><span class="n">xpath</span><span class="p">(</span><span class="s1">&#39;//div[@id=&quot;not-exists&quot;]/text()&#39;</span><span class="p">)</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="n">default</span><span class="o">=</span><span class="s1">&#39;not-found&#39;</span><span class="p">)</span>
<span class="go">&#39;not-found&#39;</span>
</pre></div>
</div>
<p>而不是使用例如 <code class="docutils literal notranslate"><span class="pre">'&#64;src'</span></code> xpath可以使用 <code class="docutils literal notranslate"><span class="pre">.attrib</span></code> A的性质 <a class="reference internal" href="#scrapy.selector.Selector" title="scrapy.selector.Selector"><code class="xref py py-class docutils literal notranslate"><span class="pre">Selector</span></code></a> ：</p>
<div class="doctest highlight-default notranslate"><div class="highlight"><pre><span></span><span class="gp">&gt;&gt;&gt; </span><span class="p">[</span><span class="n">img</span><span class="o">.</span><span class="n">attrib</span><span class="p">[</span><span class="s1">&#39;src&#39;</span><span class="p">]</span> <span class="k">for</span> <span class="n">img</span> <span class="ow">in</span> <span class="n">response</span><span class="o">.</span><span class="n">css</span><span class="p">(</span><span class="s1">&#39;img&#39;</span><span class="p">)]</span>
<span class="go">[&#39;image1_thumb.jpg&#39;,</span>
<span class="go"> &#39;image2_thumb.jpg&#39;,</span>
<span class="go"> &#39;image3_thumb.jpg&#39;,</span>
<span class="go"> &#39;image4_thumb.jpg&#39;,</span>
<span class="go"> &#39;image5_thumb.jpg&#39;]</span>
</pre></div>
</div>
<p>作为捷径， <code class="docutils literal notranslate"><span class="pre">.attrib</span></code> 也可以直接在SelectorList上使用；它返回第一个匹配元素的属性：</p>
<div class="doctest highlight-default notranslate"><div class="highlight"><pre><span></span><span class="gp">&gt;&gt;&gt; </span><span class="n">response</span><span class="o">.</span><span class="n">css</span><span class="p">(</span><span class="s1">&#39;img&#39;</span><span class="p">)</span><span class="o">.</span><span class="n">attrib</span><span class="p">[</span><span class="s1">&#39;src&#39;</span><span class="p">]</span>
<span class="go">&#39;image1_thumb.jpg&#39;</span>
</pre></div>
</div>
<p>当只需要一个结果时（例如，按id选择，或在网页上选择唯一元素时），此选项最有用：</p>
<div class="doctest highlight-default notranslate"><div class="highlight"><pre><span></span><span class="gp">&gt;&gt;&gt; </span><span class="n">response</span><span class="o">.</span><span class="n">css</span><span class="p">(</span><span class="s1">&#39;base&#39;</span><span class="p">)</span><span class="o">.</span><span class="n">attrib</span><span class="p">[</span><span class="s1">&#39;href&#39;</span><span class="p">]</span>
<span class="go">&#39;http://example.com/&#39;</span>
</pre></div>
</div>
<p>现在我们将获得基本URL和一些图像链接：</p>
<div class="doctest highlight-default notranslate"><div class="highlight"><pre><span></span><span class="gp">&gt;&gt;&gt; </span><span class="n">response</span><span class="o">.</span><span class="n">xpath</span><span class="p">(</span><span class="s1">&#39;//base/@href&#39;</span><span class="p">)</span><span class="o">.</span><span class="n">get</span><span class="p">()</span>
<span class="go">&#39;http://example.com/&#39;</span>
</pre></div>
</div>
<div class="doctest highlight-default notranslate"><div class="highlight"><pre><span></span><span class="gp">&gt;&gt;&gt; </span><span class="n">response</span><span class="o">.</span><span class="n">css</span><span class="p">(</span><span class="s1">&#39;base::attr(href)&#39;</span><span class="p">)</span><span class="o">.</span><span class="n">get</span><span class="p">()</span>
<span class="go">&#39;http://example.com/&#39;</span>
</pre></div>
</div>
<div class="doctest highlight-default notranslate"><div class="highlight"><pre><span></span><span class="gp">&gt;&gt;&gt; </span><span class="n">response</span><span class="o">.</span><span class="n">css</span><span class="p">(</span><span class="s1">&#39;base&#39;</span><span class="p">)</span><span class="o">.</span><span class="n">attrib</span><span class="p">[</span><span class="s1">&#39;href&#39;</span><span class="p">]</span>
<span class="go">&#39;http://example.com/&#39;</span>
</pre></div>
</div>
<div class="doctest highlight-default notranslate"><div class="highlight"><pre><span></span><span class="gp">&gt;&gt;&gt; </span><span class="n">response</span><span class="o">.</span><span class="n">xpath</span><span class="p">(</span><span class="s1">&#39;//a[contains(@href, &quot;image&quot;)]/@href&#39;</span><span class="p">)</span><span class="o">.</span><span class="n">getall</span><span class="p">()</span>
<span class="go">[&#39;image1.html&#39;,</span>
<span class="go"> &#39;image2.html&#39;,</span>
<span class="go"> &#39;image3.html&#39;,</span>
<span class="go"> &#39;image4.html&#39;,</span>
<span class="go"> &#39;image5.html&#39;]</span>
</pre></div>
</div>
<div class="doctest highlight-default notranslate"><div class="highlight"><pre><span></span><span class="gp">&gt;&gt;&gt; </span><span class="n">response</span><span class="o">.</span><span class="n">css</span><span class="p">(</span><span class="s1">&#39;a[href*=image]::attr(href)&#39;</span><span class="p">)</span><span class="o">.</span><span class="n">getall</span><span class="p">()</span>
<span class="go">[&#39;image1.html&#39;,</span>
<span class="go"> &#39;image2.html&#39;,</span>
<span class="go"> &#39;image3.html&#39;,</span>
<span class="go"> &#39;image4.html&#39;,</span>
<span class="go"> &#39;image5.html&#39;]</span>
</pre></div>
</div>
<div class="doctest highlight-default notranslate"><div class="highlight"><pre><span></span><span class="gp">&gt;&gt;&gt; </span><span class="n">response</span><span class="o">.</span><span class="n">xpath</span><span class="p">(</span><span class="s1">&#39;//a[contains(@href, &quot;image&quot;)]/img/@src&#39;</span><span class="p">)</span><span class="o">.</span><span class="n">getall</span><span class="p">()</span>
<span class="go">[&#39;image1_thumb.jpg&#39;,</span>
<span class="go"> &#39;image2_thumb.jpg&#39;,</span>
<span class="go"> &#39;image3_thumb.jpg&#39;,</span>
<span class="go"> &#39;image4_thumb.jpg&#39;,</span>
<span class="go"> &#39;image5_thumb.jpg&#39;]</span>
</pre></div>
</div>
<div class="doctest highlight-default notranslate"><div class="highlight"><pre><span></span><span class="gp">&gt;&gt;&gt; </span><span class="n">response</span><span class="o">.</span><span class="n">css</span><span class="p">(</span><span class="s1">&#39;a[href*=image] img::attr(src)&#39;</span><span class="p">)</span><span class="o">.</span><span class="n">getall</span><span class="p">()</span>
<span class="go">[&#39;image1_thumb.jpg&#39;,</span>
<span class="go"> &#39;image2_thumb.jpg&#39;,</span>
<span class="go"> &#39;image3_thumb.jpg&#39;,</span>
<span class="go"> &#39;image4_thumb.jpg&#39;,</span>
<span class="go"> &#39;image5_thumb.jpg&#39;]</span>
</pre></div>
</div>
</div>
<div class="section" id="extensions-to-css-selectors">
<span id="topics-selectors-css-extensions"></span><h3>CSS选择器的扩展<a class="headerlink" href="#extensions-to-css-selectors" title="永久链接至标题">¶</a></h3>
<p>根据W3C标准， <a class="reference external" href="https://www.w3.org/TR/selectors-3/#selectors">CSS selectors</a> 不支持选择文本节点或属性值。但是在Web抓取上下文中选择这些是非常重要的，以至于scrappy（parsel）实现了 <strong>non-standard pseudo-elements</strong> ：</p>
<ul class="simple">
<li><p>要选择文本节点，请使用 <code class="docutils literal notranslate"><span class="pre">::text</span></code></p></li>
<li><p>要选择属性值，请使用 <code class="docutils literal notranslate"><span class="pre">::attr(name)</span></code> 在哪里？ <em>name</em> 是要为其值的属性的名称</p></li>
</ul>
<div class="admonition warning">
<p class="admonition-title">警告</p>
<p>这些伪元素是特定于scrapy-/parsel的。他们很可能不会与其他类库合作 <a class="reference external" href="https://lxml.de/">lxml</a> 或 <a class="reference external" href="https://pypi.org/project/pyquery/">PyQuery</a> .</p>
</div>
<p>实例：</p>
<ul class="simple">
<li><p><code class="docutils literal notranslate"><span class="pre">title::text</span></code> 选择子代的子文本节点 <code class="docutils literal notranslate"><span class="pre">&lt;title&gt;</span></code> 元素：</p></li>
</ul>
<div class="doctest highlight-default notranslate"><div class="highlight"><pre><span></span><span class="gp">&gt;&gt;&gt; </span><span class="n">response</span><span class="o">.</span><span class="n">css</span><span class="p">(</span><span class="s1">&#39;title::text&#39;</span><span class="p">)</span><span class="o">.</span><span class="n">get</span><span class="p">()</span>
<span class="go">&#39;Example website&#39;</span>
</pre></div>
</div>
<ul class="simple">
<li><p><code class="docutils literal notranslate"><span class="pre">*::text</span></code> 选择当前选择器上下文的所有子代文本节点：</p></li>
</ul>
<div class="doctest highlight-default notranslate"><div class="highlight"><pre><span></span><span class="gp">&gt;&gt;&gt; </span><span class="n">response</span><span class="o">.</span><span class="n">css</span><span class="p">(</span><span class="s1">&#39;#images *::text&#39;</span><span class="p">)</span><span class="o">.</span><span class="n">getall</span><span class="p">()</span>
<span class="go">[&#39;\n   &#39;,</span>
<span class="go"> &#39;Name: My image 1 &#39;,</span>
<span class="go"> &#39;\n   &#39;,</span>
<span class="go"> &#39;Name: My image 2 &#39;,</span>
<span class="go"> &#39;\n   &#39;,</span>
<span class="go"> &#39;Name: My image 3 &#39;,</span>
<span class="go"> &#39;\n   &#39;,</span>
<span class="go"> &#39;Name: My image 4 &#39;,</span>
<span class="go"> &#39;\n   &#39;,</span>
<span class="go"> &#39;Name: My image 5 &#39;,</span>
<span class="go"> &#39;\n  &#39;]</span>
</pre></div>
</div>
<ul class="simple">
<li><p><code class="docutils literal notranslate"><span class="pre">foo::text</span></code> 如果 <code class="docutils literal notranslate"><span class="pre">foo</span></code> 元素存在，但不包含文本（即文本为空）：</p></li>
</ul>
<div class="doctest highlight-default notranslate"><div class="highlight"><pre><span></span><span class="gp">&gt;&gt;&gt; </span><span class="n">response</span><span class="o">.</span><span class="n">css</span><span class="p">(</span><span class="s1">&#39;img::text&#39;</span><span class="p">)</span><span class="o">.</span><span class="n">getall</span><span class="p">()</span>
<span class="go">[]</span>
</pre></div>
</div>
<blockquote>
<div><p>这意味着 <code class="docutils literal notranslate"><span class="pre">.css('foo::text').get()</span></code> 即使元素存在，也无法返回“无”。使用 <code class="docutils literal notranslate"><span class="pre">default=''</span></code> 如果您总是想要字符串：</p>
</div></blockquote>
<div class="doctest highlight-default notranslate"><div class="highlight"><pre><span></span><span class="gp">&gt;&gt;&gt; </span><span class="n">response</span><span class="o">.</span><span class="n">css</span><span class="p">(</span><span class="s1">&#39;img::text&#39;</span><span class="p">)</span><span class="o">.</span><span class="n">get</span><span class="p">()</span>
<span class="gp">&gt;&gt;&gt; </span><span class="n">response</span><span class="o">.</span><span class="n">css</span><span class="p">(</span><span class="s1">&#39;img::text&#39;</span><span class="p">)</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="n">default</span><span class="o">=</span><span class="s1">&#39;&#39;</span><span class="p">)</span>
<span class="go">&#39;&#39;</span>
</pre></div>
</div>
<ul class="simple">
<li><p><code class="docutils literal notranslate"><span class="pre">a::attr(href)</span></code> 选择 <em>href</em> 子链接的属性值：</p></li>
</ul>
<div class="doctest highlight-default notranslate"><div class="highlight"><pre><span></span><span class="gp">&gt;&gt;&gt; </span><span class="n">response</span><span class="o">.</span><span class="n">css</span><span class="p">(</span><span class="s1">&#39;a::attr(href)&#39;</span><span class="p">)</span><span class="o">.</span><span class="n">getall</span><span class="p">()</span>
<span class="go">[&#39;image1.html&#39;,</span>
<span class="go"> &#39;image2.html&#39;,</span>
<span class="go"> &#39;image3.html&#39;,</span>
<span class="go"> &#39;image4.html&#39;,</span>
<span class="go"> &#39;image5.html&#39;]</span>
</pre></div>
</div>
<div class="admonition note">
<p class="admonition-title">注解</p>
<p>参见： <a class="reference internal" href="#selecting-attributes"><span class="std std-ref">选择元素属性</span></a> .</p>
</div>
<div class="admonition note">
<p class="admonition-title">注解</p>
<p>不能链接这些伪元素。但在实践中，这没有多大意义：文本节点没有属性，属性值已经是字符串值，也没有子节点。</p>
</div>
</div>
<div class="section" id="nesting-selectors">
<span id="topics-selectors-nesting-selectors"></span><h3>嵌套选择器<a class="headerlink" href="#nesting-selectors" title="永久链接至标题">¶</a></h3>
<p>选择方法 (<code class="docutils literal notranslate"><span class="pre">.xpath()</span></code> 或 <code class="docutils literal notranslate"><span class="pre">.css()</span></code> )返回同一类型选择器的列表，以便您也可以调用这些选择器的选择方法。下面是一个例子：</p>
<div class="doctest highlight-default notranslate"><div class="highlight"><pre><span></span><span class="gp">&gt;&gt;&gt; </span><span class="n">links</span> <span class="o">=</span> <span class="n">response</span><span class="o">.</span><span class="n">xpath</span><span class="p">(</span><span class="s1">&#39;//a[contains(@href, &quot;image&quot;)]&#39;</span><span class="p">)</span>
<span class="gp">&gt;&gt;&gt; </span><span class="n">links</span><span class="o">.</span><span class="n">getall</span><span class="p">()</span>
<span class="go">[&#39;&lt;a href=&quot;image1.html&quot;&gt;Name: My image 1 &lt;br&gt;&lt;img src=&quot;image1_thumb.jpg&quot;&gt;&lt;/a&gt;&#39;,</span>
<span class="go"> &#39;&lt;a href=&quot;image2.html&quot;&gt;Name: My image 2 &lt;br&gt;&lt;img src=&quot;image2_thumb.jpg&quot;&gt;&lt;/a&gt;&#39;,</span>
<span class="go"> &#39;&lt;a href=&quot;image3.html&quot;&gt;Name: My image 3 &lt;br&gt;&lt;img src=&quot;image3_thumb.jpg&quot;&gt;&lt;/a&gt;&#39;,</span>
<span class="go"> &#39;&lt;a href=&quot;image4.html&quot;&gt;Name: My image 4 &lt;br&gt;&lt;img src=&quot;image4_thumb.jpg&quot;&gt;&lt;/a&gt;&#39;,</span>
<span class="go"> &#39;&lt;a href=&quot;image5.html&quot;&gt;Name: My image 5 &lt;br&gt;&lt;img src=&quot;image5_thumb.jpg&quot;&gt;&lt;/a&gt;&#39;]</span>
</pre></div>
</div>
<div class="doctest highlight-default notranslate"><div class="highlight"><pre><span></span><span class="gp">&gt;&gt;&gt; </span><span class="k">for</span> <span class="n">index</span><span class="p">,</span> <span class="n">link</span> <span class="ow">in</span> <span class="nb">enumerate</span><span class="p">(</span><span class="n">links</span><span class="p">):</span>
<span class="gp">... </span>    <span class="n">href_xpath</span> <span class="o">=</span> <span class="n">link</span><span class="o">.</span><span class="n">xpath</span><span class="p">(</span><span class="s1">&#39;@href&#39;</span><span class="p">)</span><span class="o">.</span><span class="n">get</span><span class="p">()</span>
<span class="gp">... </span>    <span class="n">img_xpath</span> <span class="o">=</span> <span class="n">link</span><span class="o">.</span><span class="n">xpath</span><span class="p">(</span><span class="s1">&#39;img/@src&#39;</span><span class="p">)</span><span class="o">.</span><span class="n">get</span><span class="p">()</span>
<span class="gp">... </span>    <span class="nb">print</span><span class="p">(</span><span class="sa">f</span><span class="s1">&#39;Link number </span><span class="si">{</span><span class="n">index</span><span class="si">}</span><span class="s1"> points to url </span><span class="si">{</span><span class="n">href_xpath</span><span class="si">!r}</span><span class="s1"> and image </span><span class="si">{</span><span class="n">img_xpath</span><span class="si">!r}</span><span class="s1">&#39;</span><span class="p">)</span>
<span class="go">Link number 0 points to url &#39;image1.html&#39; and image &#39;image1_thumb.jpg&#39;</span>
<span class="go">Link number 1 points to url &#39;image2.html&#39; and image &#39;image2_thumb.jpg&#39;</span>
<span class="go">Link number 2 points to url &#39;image3.html&#39; and image &#39;image3_thumb.jpg&#39;</span>
<span class="go">Link number 3 points to url &#39;image4.html&#39; and image &#39;image4_thumb.jpg&#39;</span>
<span class="go">Link number 4 points to url &#39;image5.html&#39; and image &#39;image5_thumb.jpg&#39;</span>
</pre></div>
</div>
</div>
<div class="section" id="selecting-element-attributes">
<span id="selecting-attributes"></span><h3>选择元素属性<a class="headerlink" href="#selecting-element-attributes" title="永久链接至标题">¶</a></h3>
<p>有几种方法可以获得属性的值。首先，可以使用XPath语法：</p>
<div class="doctest highlight-default notranslate"><div class="highlight"><pre><span></span><span class="gp">&gt;&gt;&gt; </span><span class="n">response</span><span class="o">.</span><span class="n">xpath</span><span class="p">(</span><span class="s2">&quot;//a/@href&quot;</span><span class="p">)</span><span class="o">.</span><span class="n">getall</span><span class="p">()</span>
<span class="go">[&#39;image1.html&#39;, &#39;image2.html&#39;, &#39;image3.html&#39;, &#39;image4.html&#39;, &#39;image5.html&#39;]</span>
</pre></div>
</div>
<p>xpath语法有几个优点：它是标准的xpath特性，并且 <code class="docutils literal notranslate"><span class="pre">&#64;attributes</span></code> 可用于xpath表达式的其他部分-例如，可以按属性值筛选。</p>
<p>scrapy还提供了对css选择器的扩展 (<code class="docutils literal notranslate"><span class="pre">::attr(...)</span></code> )它允许获取属性值：</p>
<div class="doctest highlight-default notranslate"><div class="highlight"><pre><span></span><span class="gp">&gt;&gt;&gt; </span><span class="n">response</span><span class="o">.</span><span class="n">css</span><span class="p">(</span><span class="s1">&#39;a::attr(href)&#39;</span><span class="p">)</span><span class="o">.</span><span class="n">getall</span><span class="p">()</span>
<span class="go">[&#39;image1.html&#39;, &#39;image2.html&#39;, &#39;image3.html&#39;, &#39;image4.html&#39;, &#39;image5.html&#39;]</span>
</pre></div>
</div>
<p>除此之外，还有 <code class="docutils literal notranslate"><span class="pre">.attrib</span></code> 选择器的属性。如果您喜欢在Python代码中查找属性，而不使用xpath或CSS扩展，则可以使用它：</p>
<div class="doctest highlight-default notranslate"><div class="highlight"><pre><span></span><span class="gp">&gt;&gt;&gt; </span><span class="p">[</span><span class="n">a</span><span class="o">.</span><span class="n">attrib</span><span class="p">[</span><span class="s1">&#39;href&#39;</span><span class="p">]</span> <span class="k">for</span> <span class="n">a</span> <span class="ow">in</span> <span class="n">response</span><span class="o">.</span><span class="n">css</span><span class="p">(</span><span class="s1">&#39;a&#39;</span><span class="p">)]</span>
<span class="go">[&#39;image1.html&#39;, &#39;image2.html&#39;, &#39;image3.html&#39;, &#39;image4.html&#39;, &#39;image5.html&#39;]</span>
</pre></div>
</div>
<p>此属性在SelectorList上也可用；它返回一个字典，其中包含第一个匹配元素的属性。当选择器预期给出单个结果时（例如，当按元素ID选择时，或在页面上选择唯一元素时），使用它非常方便：</p>
<div class="doctest highlight-default notranslate"><div class="highlight"><pre><span></span><span class="gp">&gt;&gt;&gt; </span><span class="n">response</span><span class="o">.</span><span class="n">css</span><span class="p">(</span><span class="s1">&#39;base&#39;</span><span class="p">)</span><span class="o">.</span><span class="n">attrib</span>
<span class="go">{&#39;href&#39;: &#39;http://example.com/&#39;}</span>
<span class="gp">&gt;&gt;&gt; </span><span class="n">response</span><span class="o">.</span><span class="n">css</span><span class="p">(</span><span class="s1">&#39;base&#39;</span><span class="p">)</span><span class="o">.</span><span class="n">attrib</span><span class="p">[</span><span class="s1">&#39;href&#39;</span><span class="p">]</span>
<span class="go">&#39;http://example.com/&#39;</span>
</pre></div>
</div>
<p><code class="docutils literal notranslate"><span class="pre">.attrib</span></code> 空SelectorList的属性为空：</p>
<div class="doctest highlight-default notranslate"><div class="highlight"><pre><span></span><span class="gp">&gt;&gt;&gt; </span><span class="n">response</span><span class="o">.</span><span class="n">css</span><span class="p">(</span><span class="s1">&#39;foo&#39;</span><span class="p">)</span><span class="o">.</span><span class="n">attrib</span>
<span class="go">{}</span>
</pre></div>
</div>
</div>
<div class="section" id="using-selectors-with-regular-expressions">
<h3>将选择器与正则表达式一起使用<a class="headerlink" href="#using-selectors-with-regular-expressions" title="永久链接至标题">¶</a></h3>
<p><a class="reference internal" href="#scrapy.selector.Selector" title="scrapy.selector.Selector"><code class="xref py py-class docutils literal notranslate"><span class="pre">Selector</span></code></a> 也有 <code class="docutils literal notranslate"><span class="pre">.re()</span></code> 使用正则表达式提取数据的方法。但是，与使用不同 <code class="docutils literal notranslate"><span class="pre">.xpath()</span></code> 或 <code class="docutils literal notranslate"><span class="pre">.css()</span></code> 方法， <code class="docutils literal notranslate"><span class="pre">.re()</span></code> 返回字符串列表。所以你不能构造嵌套的 <code class="docutils literal notranslate"><span class="pre">.re()</span></code> 电话。</p>
<p>下面是一个用于从 <a class="reference internal" href="#topics-selectors-htmlcode"><span class="std std-ref">HTML code</span></a> 以上：</p>
<div class="doctest highlight-default notranslate"><div class="highlight"><pre><span></span><span class="gp">&gt;&gt;&gt; </span><span class="n">response</span><span class="o">.</span><span class="n">xpath</span><span class="p">(</span><span class="s1">&#39;//a[contains(@href, &quot;image&quot;)]/text()&#39;</span><span class="p">)</span><span class="o">.</span><span class="n">re</span><span class="p">(</span><span class="sa">r</span><span class="s1">&#39;Name:\s*(.*)&#39;</span><span class="p">)</span>
<span class="go">[&#39;My image 1&#39;,</span>
<span class="go"> &#39;My image 2&#39;,</span>
<span class="go"> &#39;My image 3&#39;,</span>
<span class="go"> &#39;My image 4&#39;,</span>
<span class="go"> &#39;My image 5&#39;]</span>
</pre></div>
</div>
<p>另外还有一个助手在做往复运动 <code class="docutils literal notranslate"><span class="pre">.get()</span></code> （及其别名） <code class="docutils literal notranslate"><span class="pre">.extract_first()</span></code> 为 <code class="docutils literal notranslate"><span class="pre">.re()</span></code> 命名 <code class="docutils literal notranslate"><span class="pre">.re_first()</span></code> . 使用它只提取第一个匹配字符串：</p>
<div class="doctest highlight-default notranslate"><div class="highlight"><pre><span></span><span class="gp">&gt;&gt;&gt; </span><span class="n">response</span><span class="o">.</span><span class="n">xpath</span><span class="p">(</span><span class="s1">&#39;//a[contains(@href, &quot;image&quot;)]/text()&#39;</span><span class="p">)</span><span class="o">.</span><span class="n">re_first</span><span class="p">(</span><span class="sa">r</span><span class="s1">&#39;Name:\s*(.*)&#39;</span><span class="p">)</span>
<span class="go">&#39;My image 1&#39;</span>
</pre></div>
</div>
</div>
<div class="section" id="extract-and-extract-first">
<span id="old-extraction-api"></span><h3>extract（）和extract_first（）。<a class="headerlink" href="#extract-and-extract-first" title="永久链接至标题">¶</a></h3>
<p>如果你是一个长期的垃圾用户，你可能熟悉 <code class="docutils literal notranslate"><span class="pre">.extract()</span></code> 和 <code class="docutils literal notranslate"><span class="pre">.extract_first()</span></code> 选择器方法。许多博客文章和教程也在使用它们。这些方法仍然由Scrapy支持，有 <strong>没有计划</strong> 去贬低他们。</p>
<p>但是，现在使用 <code class="docutils literal notranslate"><span class="pre">.get()</span></code> 和 <code class="docutils literal notranslate"><span class="pre">.getall()</span></code> 方法。我们认为这些新方法会产生更简洁和可读的代码。</p>
<p>下面的例子展示了这些方法如何相互映射。</p>
<ol class="arabic">
<li><p><code class="docutils literal notranslate"><span class="pre">SelectorList.get()</span></code> 是一样的 <code class="docutils literal notranslate"><span class="pre">SelectorList.extract_first()</span></code> ：</p>
<div class="doctest highlight-default notranslate"><div class="highlight"><pre><span></span><span class="gp">&gt;&gt;&gt; </span><span class="n">response</span><span class="o">.</span><span class="n">css</span><span class="p">(</span><span class="s1">&#39;a::attr(href)&#39;</span><span class="p">)</span><span class="o">.</span><span class="n">get</span><span class="p">()</span>
<span class="go">&#39;image1.html&#39;</span>
<span class="gp">&gt;&gt;&gt; </span><span class="n">response</span><span class="o">.</span><span class="n">css</span><span class="p">(</span><span class="s1">&#39;a::attr(href)&#39;</span><span class="p">)</span><span class="o">.</span><span class="n">extract_first</span><span class="p">()</span>
<span class="go">&#39;image1.html&#39;</span>
</pre></div>
</div>
</li>
<li><p><code class="docutils literal notranslate"><span class="pre">SelectorList.getall()</span></code> 是一样的 <code class="docutils literal notranslate"><span class="pre">SelectorList.extract()</span></code> ：</p>
<div class="doctest highlight-default notranslate"><div class="highlight"><pre><span></span><span class="gp">&gt;&gt;&gt; </span><span class="n">response</span><span class="o">.</span><span class="n">css</span><span class="p">(</span><span class="s1">&#39;a::attr(href)&#39;</span><span class="p">)</span><span class="o">.</span><span class="n">getall</span><span class="p">()</span>
<span class="go">[&#39;image1.html&#39;, &#39;image2.html&#39;, &#39;image3.html&#39;, &#39;image4.html&#39;, &#39;image5.html&#39;]</span>
<span class="gp">&gt;&gt;&gt; </span><span class="n">response</span><span class="o">.</span><span class="n">css</span><span class="p">(</span><span class="s1">&#39;a::attr(href)&#39;</span><span class="p">)</span><span class="o">.</span><span class="n">extract</span><span class="p">()</span>
<span class="go">[&#39;image1.html&#39;, &#39;image2.html&#39;, &#39;image3.html&#39;, &#39;image4.html&#39;, &#39;image5.html&#39;]</span>
</pre></div>
</div>
</li>
<li><p><code class="docutils literal notranslate"><span class="pre">Selector.get()</span></code> 是一样的 <code class="docutils literal notranslate"><span class="pre">Selector.extract()</span></code> ：</p>
<div class="doctest highlight-default notranslate"><div class="highlight"><pre><span></span><span class="gp">&gt;&gt;&gt; </span><span class="n">response</span><span class="o">.</span><span class="n">css</span><span class="p">(</span><span class="s1">&#39;a::attr(href)&#39;</span><span class="p">)[</span><span class="mi">0</span><span class="p">]</span><span class="o">.</span><span class="n">get</span><span class="p">()</span>
<span class="go">&#39;image1.html&#39;</span>
<span class="gp">&gt;&gt;&gt; </span><span class="n">response</span><span class="o">.</span><span class="n">css</span><span class="p">(</span><span class="s1">&#39;a::attr(href)&#39;</span><span class="p">)[</span><span class="mi">0</span><span class="p">]</span><span class="o">.</span><span class="n">extract</span><span class="p">()</span>
<span class="go">&#39;image1.html&#39;</span>
</pre></div>
</div>
</li>
<li><p>为了保持一致性，还有 <code class="docutils literal notranslate"><span class="pre">Selector.getall()</span></code> ，返回一个列表：</p>
<div class="doctest highlight-default notranslate"><div class="highlight"><pre><span></span><span class="gp">&gt;&gt;&gt; </span><span class="n">response</span><span class="o">.</span><span class="n">css</span><span class="p">(</span><span class="s1">&#39;a::attr(href)&#39;</span><span class="p">)[</span><span class="mi">0</span><span class="p">]</span><span class="o">.</span><span class="n">getall</span><span class="p">()</span>
<span class="go">[&#39;image1.html&#39;]</span>
</pre></div>
</div>
</li>
</ol>
<p>所以，主要的区别在于 <code class="docutils literal notranslate"><span class="pre">.get()</span></code> 和 <code class="docutils literal notranslate"><span class="pre">.getall()</span></code> 方法更容易预测： <code class="docutils literal notranslate"><span class="pre">.get()</span></code> 总是返回单个结果， <code class="docutils literal notranslate"><span class="pre">.getall()</span></code> 始终返回所有提取结果的列表。用 <code class="docutils literal notranslate"><span class="pre">.extract()</span></code> 方法：结果是否为列表并不总是显而易见的；或者得到一个单独的结果 <code class="docutils literal notranslate"><span class="pre">.extract()</span></code> 或 <code class="docutils literal notranslate"><span class="pre">.extract_first()</span></code> 应该被调用。</p>
</div>
</div>
<div class="section" id="working-with-xpaths">
<span id="topics-selectors-xpaths"></span><h2>使用xpaths<a class="headerlink" href="#working-with-xpaths" title="永久链接至标题">¶</a></h2>
<p>下面是一些提示，可以帮助您有效地将xpath与scrapy选择器结合使用。如果您还不太熟悉xpath，可以先看看这个 <a class="reference external" href="http://www.zvon.org/comp/r/tut-XPath_1.html">XPath tutorial</a> .</p>
<div class="admonition note">
<p class="admonition-title">注解</p>
<p>一些提示是基于 <a class="reference external" href="https://blog.scrapinghub.com/2014/07/17/xpath-tips-from-the-web-scraping-trenches/">this post from ScrapingHub's blog</a> .</p>
</div>
<div class="section" id="working-with-relative-xpaths">
<span id="topics-selectors-relative-xpaths"></span><h3>使用相对路径<a class="headerlink" href="#working-with-relative-xpaths" title="永久链接至标题">¶</a></h3>
<p>请记住，如果要嵌套选择器并使用以开头的xpath <code class="docutils literal notranslate"><span class="pre">/</span></code> ，该xpath对文档是绝对的，而不是相对于 <code class="docutils literal notranslate"><span class="pre">Selector</span></code> 你是从打电话来的。</p>
<p>例如，假设您希望提取所有 <code class="docutils literal notranslate"><span class="pre">&lt;p&gt;</span></code> 内部元素 <code class="docutils literal notranslate"><span class="pre">&lt;div&gt;</span></code> 元素。首先，你会得到所有 <code class="docutils literal notranslate"><span class="pre">&lt;div&gt;</span></code> 元素：</p>
<div class="doctest highlight-default notranslate"><div class="highlight"><pre><span></span><span class="gp">&gt;&gt;&gt; </span><span class="n">divs</span> <span class="o">=</span> <span class="n">response</span><span class="o">.</span><span class="n">xpath</span><span class="p">(</span><span class="s1">&#39;//div&#39;</span><span class="p">)</span>
</pre></div>
</div>
<p>首先，您可能会尝试使用以下方法，这是错误的，因为它实际上提取了所有 <code class="docutils literal notranslate"><span class="pre">&lt;p&gt;</span></code> 文档中的元素，而不仅仅是其中的元素 <code class="docutils literal notranslate"><span class="pre">&lt;div&gt;</span></code> 元素：</p>
<div class="doctest highlight-default notranslate"><div class="highlight"><pre><span></span><span class="gp">&gt;&gt;&gt; </span><span class="k">for</span> <span class="n">p</span> <span class="ow">in</span> <span class="n">divs</span><span class="o">.</span><span class="n">xpath</span><span class="p">(</span><span class="s1">&#39;//p&#39;</span><span class="p">):</span>  <span class="c1"># this is wrong - gets all &lt;p&gt; from the whole document</span>
<span class="gp">... </span>    <span class="nb">print</span><span class="p">(</span><span class="n">p</span><span class="o">.</span><span class="n">get</span><span class="p">())</span>
</pre></div>
</div>
<p>这是正确的方法（注意在 <code class="docutils literal notranslate"><span class="pre">.//p</span></code> XPath）：</p>
<div class="doctest highlight-default notranslate"><div class="highlight"><pre><span></span><span class="gp">&gt;&gt;&gt; </span><span class="k">for</span> <span class="n">p</span> <span class="ow">in</span> <span class="n">divs</span><span class="o">.</span><span class="n">xpath</span><span class="p">(</span><span class="s1">&#39;.//p&#39;</span><span class="p">):</span>  <span class="c1"># extracts all &lt;p&gt; inside</span>
<span class="gp">... </span>    <span class="nb">print</span><span class="p">(</span><span class="n">p</span><span class="o">.</span><span class="n">get</span><span class="p">())</span>
</pre></div>
</div>
<p>另一个常见的情况是提取所有直接 <code class="docutils literal notranslate"><span class="pre">&lt;p&gt;</span></code> 儿童：</p>
<div class="doctest highlight-default notranslate"><div class="highlight"><pre><span></span><span class="gp">&gt;&gt;&gt; </span><span class="k">for</span> <span class="n">p</span> <span class="ow">in</span> <span class="n">divs</span><span class="o">.</span><span class="n">xpath</span><span class="p">(</span><span class="s1">&#39;p&#39;</span><span class="p">):</span>
<span class="gp">... </span>    <span class="nb">print</span><span class="p">(</span><span class="n">p</span><span class="o">.</span><span class="n">get</span><span class="p">())</span>
</pre></div>
</div>
<p>有关相对路径的更多详细信息，请参见 <a class="reference external" href="https://www.w3.org/TR/xpath/all/#location-paths">Location Paths</a> XPath规范中的节。</p>
</div>
<div class="section" id="when-querying-by-class-consider-using-css">
<h3>按类查询时，请考虑使用CSS<a class="headerlink" href="#when-querying-by-class-consider-using-css" title="永久链接至标题">¶</a></h3>
<p>因为一个元素可以包含多个CSS类，所以按类选择元素的xpath方法相当冗长：</p>
<div class="highlight-python notranslate"><div class="highlight"><pre><span></span><span class="o">*</span><span class="p">[</span><span class="n">contains</span><span class="p">(</span><span class="n">concat</span><span class="p">(</span><span class="s1">&#39; &#39;</span><span class="p">,</span> <span class="n">normalize</span><span class="o">-</span><span class="n">space</span><span class="p">(</span><span class="nd">@class</span><span class="p">),</span> <span class="s1">&#39; &#39;</span><span class="p">),</span> <span class="s1">&#39; someclass &#39;</span><span class="p">)]</span>
</pre></div>
</div>
<p>如果你使用 <code class="docutils literal notranslate"><span class="pre">&#64;class='someclass'</span></code> 如果只使用 <code class="docutils literal notranslate"><span class="pre">contains(&#64;class,</span> <span class="pre">'someclass')</span></code> 为了弥补这一点，如果元素具有共享字符串的不同类名，那么最终可能会得到更多想要的元素。 <code class="docutils literal notranslate"><span class="pre">someclass</span></code> .</p>
<p>事实证明，剪贴选择器允许您链接选择器，因此大多数时候，您都可以使用CSS按类选择，然后在需要时切换到XPath：</p>
<div class="doctest highlight-default notranslate"><div class="highlight"><pre><span></span><span class="gp">&gt;&gt;&gt; </span><span class="kn">from</span> <span class="nn">scrapy</span> <span class="kn">import</span> <span class="n">Selector</span>
<span class="gp">&gt;&gt;&gt; </span><span class="n">sel</span> <span class="o">=</span> <span class="n">Selector</span><span class="p">(</span><span class="n">text</span><span class="o">=</span><span class="s1">&#39;&lt;div class=&quot;hero shout&quot;&gt;&lt;time datetime=&quot;2014-07-23 19:00&quot;&gt;Special date&lt;/time&gt;&lt;/div&gt;&#39;</span><span class="p">)</span>
<span class="gp">&gt;&gt;&gt; </span><span class="n">sel</span><span class="o">.</span><span class="n">css</span><span class="p">(</span><span class="s1">&#39;.shout&#39;</span><span class="p">)</span><span class="o">.</span><span class="n">xpath</span><span class="p">(</span><span class="s1">&#39;./time/@datetime&#39;</span><span class="p">)</span><span class="o">.</span><span class="n">getall</span><span class="p">()</span>
<span class="go">[&#39;2014-07-23 19:00&#39;]</span>
</pre></div>
</div>
<p>这比使用上面显示的详细的xpath技巧要干净。只要记住使用 <code class="docutils literal notranslate"><span class="pre">.</span></code> 在后面的xpath表达式中。</p>
</div>
<div class="section" id="beware-of-the-difference-between-node-1-and-node-1">
<h3>注意//节点之间的区别 [1] 和（/ /节点） [1]<a class="headerlink" href="#beware-of-the-difference-between-node-1-and-node-1" title="永久链接至标题">¶</a></h3>
<p><code class="docutils literal notranslate"><span class="pre">//node[1]</span></code> 选择所有首先出现在各自父节点下的节点。</p>
<p><code class="docutils literal notranslate"><span class="pre">(//node)[1]</span></code> 选择文档中的所有节点，然后只获取其中的第一个节点。</p>
<p>例子：</p>
<div class="doctest highlight-default notranslate"><div class="highlight"><pre><span></span><span class="gp">&gt;&gt;&gt; </span><span class="kn">from</span> <span class="nn">scrapy</span> <span class="kn">import</span> <span class="n">Selector</span>
<span class="gp">&gt;&gt;&gt; </span><span class="n">sel</span> <span class="o">=</span> <span class="n">Selector</span><span class="p">(</span><span class="n">text</span><span class="o">=</span><span class="s2">&quot;&quot;&quot;</span>
<span class="go">....:     &lt;ul class=&quot;list&quot;&gt;</span>
<span class="go">....:         &lt;li&gt;1&lt;/li&gt;</span>
<span class="go">....:         &lt;li&gt;2&lt;/li&gt;</span>
<span class="go">....:         &lt;li&gt;3&lt;/li&gt;</span>
<span class="go">....:     &lt;/ul&gt;</span>
<span class="go">....:     &lt;ul class=&quot;list&quot;&gt;</span>
<span class="go">....:         &lt;li&gt;4&lt;/li&gt;</span>
<span class="go">....:         &lt;li&gt;5&lt;/li&gt;</span>
<span class="go">....:         &lt;li&gt;6&lt;/li&gt;</span>
<span class="go">....:     &lt;/ul&gt;&quot;&quot;&quot;)</span>
<span class="gp">&gt;&gt;&gt; </span><span class="n">xp</span> <span class="o">=</span> <span class="k">lambda</span> <span class="n">x</span><span class="p">:</span> <span class="n">sel</span><span class="o">.</span><span class="n">xpath</span><span class="p">(</span><span class="n">x</span><span class="p">)</span><span class="o">.</span><span class="n">getall</span><span class="p">()</span>
</pre></div>
</div>
<p>这是最重要的 <code class="docutils literal notranslate"><span class="pre">&lt;li&gt;</span></code> 任何父元素下的元素：</p>
<div class="doctest highlight-default notranslate"><div class="highlight"><pre><span></span><span class="gp">&gt;&gt;&gt; </span><span class="n">xp</span><span class="p">(</span><span class="s2">&quot;//li[1]&quot;</span><span class="p">)</span>
<span class="go">[&#39;&lt;li&gt;1&lt;/li&gt;&#39;, &#39;&lt;li&gt;4&lt;/li&gt;&#39;]</span>
</pre></div>
</div>
<p>这是第一个 <code class="docutils literal notranslate"><span class="pre">&lt;li&gt;</span></code> 整个文档中的元素：</p>
<div class="doctest highlight-default notranslate"><div class="highlight"><pre><span></span><span class="gp">&gt;&gt;&gt; </span><span class="n">xp</span><span class="p">(</span><span class="s2">&quot;(//li)[1]&quot;</span><span class="p">)</span>
<span class="go">[&#39;&lt;li&gt;1&lt;/li&gt;&#39;]</span>
</pre></div>
</div>
<p>这是最重要的 <code class="docutils literal notranslate"><span class="pre">&lt;li&gt;</span></code> 下的元素 <code class="docutils literal notranslate"><span class="pre">&lt;ul&gt;</span></code> 起源：</p>
<div class="doctest highlight-default notranslate"><div class="highlight"><pre><span></span><span class="gp">&gt;&gt;&gt; </span><span class="n">xp</span><span class="p">(</span><span class="s2">&quot;//ul/li[1]&quot;</span><span class="p">)</span>
<span class="go">[&#39;&lt;li&gt;1&lt;/li&gt;&#39;, &#39;&lt;li&gt;4&lt;/li&gt;&#39;]</span>
</pre></div>
</div>
<p>这是第一个 <code class="docutils literal notranslate"><span class="pre">&lt;li&gt;</span></code> 元素在 <code class="docutils literal notranslate"><span class="pre">&lt;ul&gt;</span></code> 整个文档中的父级：</p>
<div class="doctest highlight-default notranslate"><div class="highlight"><pre><span></span><span class="gp">&gt;&gt;&gt; </span><span class="n">xp</span><span class="p">(</span><span class="s2">&quot;(//ul/li)[1]&quot;</span><span class="p">)</span>
<span class="go">[&#39;&lt;li&gt;1&lt;/li&gt;&#39;]</span>
</pre></div>
</div>
</div>
<div class="section" id="using-text-nodes-in-a-condition">
<h3>在条件中使用文本节点<a class="headerlink" href="#using-text-nodes-in-a-condition" title="永久链接至标题">¶</a></h3>
<p>当需要将文本内容用作 <a class="reference external" href="https://www.w3.org/TR/xpath/all/#section-String-Functions">XPath string function</a> 避免使用 <code class="docutils literal notranslate"><span class="pre">.//text()</span></code> and use just <code class="docutils literal notranslate"><span class="pre">.</span></code> 相反。</p>
<p>这是因为表达式 <code class="docutils literal notranslate"><span class="pre">.//text()</span></code> 生成一个文本元素集合--a <em>node-set</em> . 当一个节点集被转换成一个字符串时，当它作为参数传递给一个字符串函数（如 <code class="docutils literal notranslate"><span class="pre">contains()</span></code> 或 <code class="docutils literal notranslate"><span class="pre">starts-with()</span></code> ，它只为第一个元素生成文本。</p>
<p>例子：</p>
<div class="doctest highlight-default notranslate"><div class="highlight"><pre><span></span><span class="gp">&gt;&gt;&gt; </span><span class="kn">from</span> <span class="nn">scrapy</span> <span class="kn">import</span> <span class="n">Selector</span>
<span class="gp">&gt;&gt;&gt; </span><span class="n">sel</span> <span class="o">=</span> <span class="n">Selector</span><span class="p">(</span><span class="n">text</span><span class="o">=</span><span class="s1">&#39;&lt;a href=&quot;#&quot;&gt;Click here to go to the &lt;strong&gt;Next Page&lt;/strong&gt;&lt;/a&gt;&#39;</span><span class="p">)</span>
</pre></div>
</div>
<p>转换A <em>node-set</em> 字符串：</p>
<div class="doctest highlight-default notranslate"><div class="highlight"><pre><span></span><span class="gp">&gt;&gt;&gt; </span><span class="n">sel</span><span class="o">.</span><span class="n">xpath</span><span class="p">(</span><span class="s1">&#39;//a//text()&#39;</span><span class="p">)</span><span class="o">.</span><span class="n">getall</span><span class="p">()</span> <span class="c1"># take a peek at the node-set</span>
<span class="go">[&#39;Click here to go to the &#39;, &#39;Next Page&#39;]</span>
<span class="gp">&gt;&gt;&gt; </span><span class="n">sel</span><span class="o">.</span><span class="n">xpath</span><span class="p">(</span><span class="s2">&quot;string(//a[1]//text())&quot;</span><span class="p">)</span><span class="o">.</span><span class="n">getall</span><span class="p">()</span> <span class="c1"># convert it to string</span>
<span class="go">[&#39;Click here to go to the &#39;]</span>
</pre></div>
</div>
<p>A <em>node</em> 但是，转换为字符串后，会将其自身的文本加上其所有后代的文本组合在一起：</p>
<div class="doctest highlight-default notranslate"><div class="highlight"><pre><span></span><span class="gp">&gt;&gt;&gt; </span><span class="n">sel</span><span class="o">.</span><span class="n">xpath</span><span class="p">(</span><span class="s2">&quot;//a[1]&quot;</span><span class="p">)</span><span class="o">.</span><span class="n">getall</span><span class="p">()</span> <span class="c1"># select the first node</span>
<span class="go">[&#39;&lt;a href=&quot;#&quot;&gt;Click here to go to the &lt;strong&gt;Next Page&lt;/strong&gt;&lt;/a&gt;&#39;]</span>
<span class="gp">&gt;&gt;&gt; </span><span class="n">sel</span><span class="o">.</span><span class="n">xpath</span><span class="p">(</span><span class="s2">&quot;string(//a[1])&quot;</span><span class="p">)</span><span class="o">.</span><span class="n">getall</span><span class="p">()</span> <span class="c1"># convert it to string</span>
<span class="go">[&#39;Click here to go to the Next Page&#39;]</span>
</pre></div>
</div>
<p>所以，使用 <code class="docutils literal notranslate"><span class="pre">.//text()</span></code> 在这种情况下，节点集不会选择任何内容：</p>
<div class="doctest highlight-default notranslate"><div class="highlight"><pre><span></span><span class="gp">&gt;&gt;&gt; </span><span class="n">sel</span><span class="o">.</span><span class="n">xpath</span><span class="p">(</span><span class="s2">&quot;//a[contains(.//text(), &#39;Next Page&#39;)]&quot;</span><span class="p">)</span><span class="o">.</span><span class="n">getall</span><span class="p">()</span>
<span class="go">[]</span>
</pre></div>
</div>
<p>但是使用 <code class="docutils literal notranslate"><span class="pre">.</span></code> 指的是节点：</p>
<div class="doctest highlight-default notranslate"><div class="highlight"><pre><span></span><span class="gp">&gt;&gt;&gt; </span><span class="n">sel</span><span class="o">.</span><span class="n">xpath</span><span class="p">(</span><span class="s2">&quot;//a[contains(., &#39;Next Page&#39;)]&quot;</span><span class="p">)</span><span class="o">.</span><span class="n">getall</span><span class="p">()</span>
<span class="go">[&#39;&lt;a href=&quot;#&quot;&gt;Click here to go to the &lt;strong&gt;Next Page&lt;/strong&gt;&lt;/a&gt;&#39;]</span>
</pre></div>
</div>
</div>
<div class="section" id="variables-in-xpath-expressions">
<span id="topics-selectors-xpath-variables"></span><h3>xpath表达式中的变量<a class="headerlink" href="#variables-in-xpath-expressions" title="永久链接至标题">¶</a></h3>
<p>xpath允许您引用xpath表达式中的变量，使用 <code class="docutils literal notranslate"><span class="pre">$somevariable</span></code> 语法。这与SQL世界中的参数化查询或准备好的语句有点类似，在SQL世界中，将查询中的某些参数替换为诸如 <code class="docutils literal notranslate"><span class="pre">?</span></code> ，然后用查询传递的值替换。</p>
<p>下面是一个基于元素“id”属性值匹配元素的示例，而不必对其进行硬编码（如前所示）：</p>
<div class="doctest highlight-default notranslate"><div class="highlight"><pre><span></span><span class="gp">&gt;&gt;&gt; </span><span class="c1"># `$val` used in the expression, a `val` argument needs to be passed</span>
<span class="gp">&gt;&gt;&gt; </span><span class="n">response</span><span class="o">.</span><span class="n">xpath</span><span class="p">(</span><span class="s1">&#39;//div[@id=$val]/a/text()&#39;</span><span class="p">,</span> <span class="n">val</span><span class="o">=</span><span class="s1">&#39;images&#39;</span><span class="p">)</span><span class="o">.</span><span class="n">get</span><span class="p">()</span>
<span class="go">&#39;Name: My image 1 &#39;</span>
</pre></div>
</div>
<p>下面是另一个示例，用于查找 <code class="docutils literal notranslate"><span class="pre">&lt;div&gt;</span></code> 包含五个的标签 <code class="docutils literal notranslate"><span class="pre">&lt;a&gt;</span></code> 孩子们（在这里我们传递价值 <code class="docutils literal notranslate"><span class="pre">5</span></code> 整数形式）：</p>
<div class="doctest highlight-default notranslate"><div class="highlight"><pre><span></span><span class="gp">&gt;&gt;&gt; </span><span class="n">response</span><span class="o">.</span><span class="n">xpath</span><span class="p">(</span><span class="s1">&#39;//div[count(a)=$cnt]/@id&#39;</span><span class="p">,</span> <span class="n">cnt</span><span class="o">=</span><span class="mi">5</span><span class="p">)</span><span class="o">.</span><span class="n">get</span><span class="p">()</span>
<span class="go">&#39;images&#39;</span>
</pre></div>
</div>
<p>调用时，所有变量引用都必须具有绑定值 <code class="docutils literal notranslate"><span class="pre">.xpath()</span></code> （否则你会得到 <code class="docutils literal notranslate"><span class="pre">ValueError:</span> <span class="pre">XPath</span> <span class="pre">error:</span></code> 例外）。这是通过根据需要传递尽可能多的命名参数来实现的。</p>
<p><cite>parsel</cite> _为scrappy选择器供电的库提供了更多的细节和示例 <a class="reference external" href="https://parsel.readthedocs.io/en/latest/usage.html#variables-in-xpath-expressions">XPath variables</a> .</p>
</div>
<div class="section" id="removing-namespaces">
<span id="id2"></span><h3>正在删除命名空间<a class="headerlink" href="#removing-namespaces" title="永久链接至标题">¶</a></h3>
<p>在处理 爬取  项目时，完全消除名称空间，只使用元素名，编写更简单/方便的xpaths通常是非常方便的。你可以使用 <code class="xref py py-meth docutils literal notranslate"><span class="pre">Selector.remove_namespaces()</span></code> 方法。</p>
<p>让我们展示一个例子，用Python Insider博客Atom feed来说明这一点。</p>
<p>首先，我们用要抓取的URL打开外壳：</p>
<div class="highlight-sh notranslate"><div class="highlight"><pre><span></span>$ scrapy shell https://feeds.feedburner.com/PythonInsider
</pre></div>
</div>
<p>文件就是这样开始的：</p>
<div class="highlight-sh notranslate"><div class="highlight"><pre><span></span>&lt;?xml <span class="nv">version</span><span class="o">=</span><span class="s2">&quot;1.0&quot;</span> <span class="nv">encoding</span><span class="o">=</span><span class="s2">&quot;UTF-8&quot;</span>?&gt;
&lt;?xml-stylesheet ...
&lt;feed <span class="nv">xmlns</span><span class="o">=</span><span class="s2">&quot;http://www.w3.org/2005/Atom&quot;</span>
      xmlns:openSearch<span class="o">=</span><span class="s2">&quot;http://a9.com/-/spec/opensearchrss/1.0/&quot;</span>
      xmlns:blogger<span class="o">=</span><span class="s2">&quot;http://schemas.google.com/blogger/2008&quot;</span>
      xmlns:georss<span class="o">=</span><span class="s2">&quot;http://www.georss.org/georss&quot;</span>
      xmlns:gd<span class="o">=</span><span class="s2">&quot;http://schemas.google.com/g/2005&quot;</span>
      xmlns:thr<span class="o">=</span><span class="s2">&quot;http://purl.org/syndication/thread/1.0&quot;</span>
      xmlns:feedburner<span class="o">=</span><span class="s2">&quot;http://rssnamespace.org/feedburner/ext/1.0&quot;</span>&gt;
  ...
</pre></div>
</div>
<p>您可以看到几个名称空间声明，其中包括默认的“<a class="reference external" href="http://www.w3.org/2005/atom">http://www.w3.org/2005/atom</a>”，以及另一个使用“gd:”前缀的“<a class="reference external" href="http://schemas.google.com/g/2005">http://schemas.google.com/g/2005</a>”。</p>
<p>一旦进入外壳，我们可以尝试选择所有 <code class="docutils literal notranslate"><span class="pre">&lt;link&gt;</span></code> 对象并查看它是否不起作用（因为Atom XML命名空间混淆了这些节点）：</p>
<div class="doctest highlight-default notranslate"><div class="highlight"><pre><span></span><span class="gp">&gt;&gt;&gt; </span><span class="n">response</span><span class="o">.</span><span class="n">xpath</span><span class="p">(</span><span class="s2">&quot;//link&quot;</span><span class="p">)</span>
<span class="go">[]</span>
</pre></div>
</div>
<p>但一旦我们呼叫 <code class="xref py py-meth docutils literal notranslate"><span class="pre">Selector.remove_namespaces()</span></code> 方法，则所有节点都可以通过其名称直接访问：</p>
<div class="doctest highlight-default notranslate"><div class="highlight"><pre><span></span><span class="gp">&gt;&gt;&gt; </span><span class="n">response</span><span class="o">.</span><span class="n">selector</span><span class="o">.</span><span class="n">remove_namespaces</span><span class="p">()</span>
<span class="gp">&gt;&gt;&gt; </span><span class="n">response</span><span class="o">.</span><span class="n">xpath</span><span class="p">(</span><span class="s2">&quot;//link&quot;</span><span class="p">)</span>
<span class="go">[&lt;Selector xpath=&#39;//link&#39; data=&#39;&lt;link rel=&quot;alternate&quot; type=&quot;text/html&quot; h&#39;&gt;,</span>
<span class="go">    &lt;Selector xpath=&#39;//link&#39; data=&#39;&lt;link rel=&quot;next&quot; type=&quot;application/atom+&#39;&gt;,</span>
<span class="go">    ...</span>
</pre></div>
</div>
<p>如果您想知道为什么不总是在默认情况下调用命名空间移除过程，而不必手动调用它，这是因为两个原因，按照相关性的顺序，这是：</p>
<ol class="arabic simple">
<li><p>删除命名空间需要迭代和修改文档中的所有节点，这是一个相当昂贵的操作，默认情况下，对scrapy所爬行的所有文档执行此操作。</p></li>
<li><p>在某些情况下，实际需要使用名称空间，以防某些元素名称在名称空间之间发生冲突。不过，这些病例非常罕见。</p></li>
</ol>
</div>
<div class="section" id="using-exslt-extensions">
<h3>使用exslt扩展<a class="headerlink" href="#using-exslt-extensions" title="永久链接至标题">¶</a></h3>
<p>建在顶上 <a class="reference external" href="https://lxml.de/">lxml</a> ，scrapy选择器支持一些 <a class="reference external" href="http://exslt.org/">EXSLT</a> 扩展名，并附带这些预注册的命名空间以在xpath表达式中使用：</p>
<table class="docutils align-default">
<colgroup>
<col style="width: 9%" />
<col style="width: 56%" />
<col style="width: 35%" />
</colgroup>
<thead>
<tr class="row-odd"><th class="head"><p>前缀</p></th>
<th class="head"><p>命名空间</p></th>
<th class="head"><p>使用</p></th>
</tr>
</thead>
<tbody>
<tr class="row-even"><td><p>重新</p></td>
<td><p><a class="reference external" href="http://exslt.org">http://exslt.org</a>/正则表达式</p></td>
<td><p><a class="reference external" href="http://exslt.org/regexp/index.html">regular expressions</a></p></td>
</tr>
<tr class="row-odd"><td><p>设置</p></td>
<td><p>http://exslt.org/sets</p></td>
<td><p><a class="reference external" href="http://exslt.org/set/index.html">set manipulation</a></p></td>
</tr>
</tbody>
</table>
<div class="section" id="regular-expressions">
<h4>正则表达式<a class="headerlink" href="#regular-expressions" title="永久链接至标题">¶</a></h4>
<p>这个 <code class="docutils literal notranslate"><span class="pre">test()</span></code> 例如，当xpath的 <code class="docutils literal notranslate"><span class="pre">starts-with()</span></code> 或 <code class="docutils literal notranslate"><span class="pre">contains()</span></code> 还不够。</p>
<p>在列表项中选择“class”属性以数字结尾的链接的示例：</p>
<div class="doctest highlight-default notranslate"><div class="highlight"><pre><span></span><span class="gp">&gt;&gt;&gt; </span><span class="kn">from</span> <span class="nn">scrapy</span> <span class="kn">import</span> <span class="n">Selector</span>
<span class="gp">&gt;&gt;&gt; </span><span class="n">doc</span> <span class="o">=</span> <span class="s2">&quot;&quot;&quot;</span>
<span class="gp">... </span><span class="s2">&lt;div&gt;</span>
<span class="gp">... </span><span class="s2">    &lt;ul&gt;</span>
<span class="gp">... </span><span class="s2">        &lt;li class=&quot;item-0&quot;&gt;&lt;a href=&quot;link1.html&quot;&gt;first item&lt;/a&gt;&lt;/li&gt;</span>
<span class="gp">... </span><span class="s2">        &lt;li class=&quot;item-1&quot;&gt;&lt;a href=&quot;link2.html&quot;&gt;second item&lt;/a&gt;&lt;/li&gt;</span>
<span class="gp">... </span><span class="s2">        &lt;li class=&quot;item-inactive&quot;&gt;&lt;a href=&quot;link3.html&quot;&gt;third item&lt;/a&gt;&lt;/li&gt;</span>
<span class="gp">... </span><span class="s2">        &lt;li class=&quot;item-1&quot;&gt;&lt;a href=&quot;link4.html&quot;&gt;fourth item&lt;/a&gt;&lt;/li&gt;</span>
<span class="gp">... </span><span class="s2">        &lt;li class=&quot;item-0&quot;&gt;&lt;a href=&quot;link5.html&quot;&gt;fifth item&lt;/a&gt;&lt;/li&gt;</span>
<span class="gp">... </span><span class="s2">    &lt;/ul&gt;</span>
<span class="gp">... </span><span class="s2">&lt;/div&gt;</span>
<span class="gp">... </span><span class="s2">&quot;&quot;&quot;</span>
<span class="gp">&gt;&gt;&gt; </span><span class="n">sel</span> <span class="o">=</span> <span class="n">Selector</span><span class="p">(</span><span class="n">text</span><span class="o">=</span><span class="n">doc</span><span class="p">,</span> <span class="nb">type</span><span class="o">=</span><span class="s2">&quot;html&quot;</span><span class="p">)</span>
<span class="gp">&gt;&gt;&gt; </span><span class="n">sel</span><span class="o">.</span><span class="n">xpath</span><span class="p">(</span><span class="s1">&#39;//li//@href&#39;</span><span class="p">)</span><span class="o">.</span><span class="n">getall</span><span class="p">()</span>
<span class="go">[&#39;link1.html&#39;, &#39;link2.html&#39;, &#39;link3.html&#39;, &#39;link4.html&#39;, &#39;link5.html&#39;]</span>
<span class="gp">&gt;&gt;&gt; </span><span class="n">sel</span><span class="o">.</span><span class="n">xpath</span><span class="p">(</span><span class="s1">&#39;//li[re:test(@class, &quot;item-\d$&quot;)]//@href&#39;</span><span class="p">)</span><span class="o">.</span><span class="n">getall</span><span class="p">()</span>
<span class="go">[&#39;link1.html&#39;, &#39;link2.html&#39;, &#39;link4.html&#39;, &#39;link5.html&#39;]</span>
</pre></div>
</div>
<div class="admonition warning">
<p class="admonition-title">警告</p>
<p>C库 <code class="docutils literal notranslate"><span class="pre">libxslt</span></code> 本机不支持exslt正则表达式，因此 <a class="reference external" href="https://lxml.de/">lxml</a> 的实现使用了对python的钩子 <code class="docutils literal notranslate"><span class="pre">re</span></code> 模块。因此，在xpath表达式中使用regexp函数可能会增加一点性能损失。</p>
</div>
</div>
<div class="section" id="set-operations">
<h4>集合运算<a class="headerlink" href="#set-operations" title="永久链接至标题">¶</a></h4>
<p>例如，在提取文本元素之前，可以方便地排除文档树的某些部分。</p>
<p>示例提取微数据（样本内容取自https://schema.org/Product)包含一组itemscope和相应的itemprop:：</p>
<div class="highlight-python notranslate"><div class="highlight"><pre><span></span><span class="gp">&gt;&gt;&gt; </span><span class="n">doc</span> <span class="o">=</span> <span class="s2">&quot;&quot;&quot;</span>
<span class="gp">... </span><span class="s2">&lt;div itemscope itemtype=&quot;http://schema.org/Product&quot;&gt;</span>
<span class="gp">... </span><span class="s2">  &lt;span itemprop=&quot;name&quot;&gt;Kenmore White 17&quot; Microwave&lt;/span&gt;</span>
<span class="gp">... </span><span class="s2">  &lt;img src=&quot;kenmore-microwave-17in.jpg&quot; alt=&#39;Kenmore 17&quot; Microwave&#39; /&gt;</span>
<span class="gp">... </span><span class="s2">  &lt;div itemprop=&quot;aggregateRating&quot;</span>
<span class="gp">... </span><span class="s2">    itemscope itemtype=&quot;http://schema.org/AggregateRating&quot;&gt;</span>
<span class="gp">... </span><span class="s2">   Rated &lt;span itemprop=&quot;ratingValue&quot;&gt;3.5&lt;/span&gt;/5</span>
<span class="gp">... </span><span class="s2">   based on &lt;span itemprop=&quot;reviewCount&quot;&gt;11&lt;/span&gt; customer reviews</span>
<span class="gp">... </span><span class="s2">  &lt;/div&gt;</span>
<span class="gp">...</span><span class="s2"></span>
<span class="gp">... </span><span class="s2">  &lt;div itemprop=&quot;offers&quot; itemscope itemtype=&quot;http://schema.org/Offer&quot;&gt;</span>
<span class="gp">... </span><span class="s2">    &lt;span itemprop=&quot;price&quot;&gt;$55.00&lt;/span&gt;</span>
<span class="gp">... </span><span class="s2">    &lt;link itemprop=&quot;availability&quot; href=&quot;http://schema.org/InStock&quot; /&gt;In stock</span>
<span class="gp">... </span><span class="s2">  &lt;/div&gt;</span>
<span class="gp">...</span><span class="s2"></span>
<span class="gp">... </span><span class="s2">  Product description:</span>
<span class="gp">... </span><span class="s2">  &lt;span itemprop=&quot;description&quot;&gt;0.7 cubic feet countertop microwave.</span>
<span class="gp">... </span><span class="s2">  Has six preset cooking categories and convenience features like</span>
<span class="gp">... </span><span class="s2">  Add-A-Minute and Child Lock.&lt;/span&gt;</span>
<span class="gp">...</span><span class="s2"></span>
<span class="gp">... </span><span class="s2">  Customer reviews:</span>
<span class="gp">...</span><span class="s2"></span>
<span class="gp">... </span><span class="s2">  &lt;div itemprop=&quot;review&quot; itemscope itemtype=&quot;http://schema.org/Review&quot;&gt;</span>
<span class="gp">... </span><span class="s2">    &lt;span itemprop=&quot;name&quot;&gt;Not a happy camper&lt;/span&gt; -</span>
<span class="gp">... </span><span class="s2">    by &lt;span itemprop=&quot;author&quot;&gt;Ellie&lt;/span&gt;,</span>
<span class="gp">... </span><span class="s2">    &lt;meta itemprop=&quot;datePublished&quot; content=&quot;2011-04-01&quot;&gt;April 1, 2011</span>
<span class="gp">... </span><span class="s2">    &lt;div itemprop=&quot;reviewRating&quot; itemscope itemtype=&quot;http://schema.org/Rating&quot;&gt;</span>
<span class="gp">... </span><span class="s2">      &lt;meta itemprop=&quot;worstRating&quot; content = &quot;1&quot;&gt;</span>
<span class="gp">... </span><span class="s2">      &lt;span itemprop=&quot;ratingValue&quot;&gt;1&lt;/span&gt;/</span>
<span class="gp">... </span><span class="s2">      &lt;span itemprop=&quot;bestRating&quot;&gt;5&lt;/span&gt;stars</span>
<span class="gp">... </span><span class="s2">    &lt;/div&gt;</span>
<span class="gp">... </span><span class="s2">    &lt;span itemprop=&quot;description&quot;&gt;The lamp burned out and now I have to replace</span>
<span class="gp">... </span><span class="s2">    it. &lt;/span&gt;</span>
<span class="gp">... </span><span class="s2">  &lt;/div&gt;</span>
<span class="gp">...</span><span class="s2"></span>
<span class="gp">... </span><span class="s2">  &lt;div itemprop=&quot;review&quot; itemscope itemtype=&quot;http://schema.org/Review&quot;&gt;</span>
<span class="gp">... </span><span class="s2">    &lt;span itemprop=&quot;name&quot;&gt;Value purchase&lt;/span&gt; -</span>
<span class="gp">... </span><span class="s2">    by &lt;span itemprop=&quot;author&quot;&gt;Lucas&lt;/span&gt;,</span>
<span class="gp">... </span><span class="s2">    &lt;meta itemprop=&quot;datePublished&quot; content=&quot;2011-03-25&quot;&gt;March 25, 2011</span>
<span class="gp">... </span><span class="s2">    &lt;div itemprop=&quot;reviewRating&quot; itemscope itemtype=&quot;http://schema.org/Rating&quot;&gt;</span>
<span class="gp">... </span><span class="s2">      &lt;meta itemprop=&quot;worstRating&quot; content = &quot;1&quot;/&gt;</span>
<span class="gp">... </span><span class="s2">      &lt;span itemprop=&quot;ratingValue&quot;&gt;4&lt;/span&gt;/</span>
<span class="gp">... </span><span class="s2">      &lt;span itemprop=&quot;bestRating&quot;&gt;5&lt;/span&gt;stars</span>
<span class="gp">... </span><span class="s2">    &lt;/div&gt;</span>
<span class="gp">... </span><span class="s2">    &lt;span itemprop=&quot;description&quot;&gt;Great microwave for the price. It is small and</span>
<span class="gp">... </span><span class="s2">    fits in my apartment.&lt;/span&gt;</span>
<span class="gp">... </span><span class="s2">  &lt;/div&gt;</span>
<span class="gp">... </span><span class="s2">  ...</span>
<span class="gp">... </span><span class="s2">&lt;/div&gt;</span>
<span class="gp">... </span><span class="s2">&quot;&quot;&quot;</span>
<span class="gp">&gt;&gt;&gt; </span><span class="n">sel</span> <span class="o">=</span> <span class="n">Selector</span><span class="p">(</span><span class="n">text</span><span class="o">=</span><span class="n">doc</span><span class="p">,</span> <span class="nb">type</span><span class="o">=</span><span class="s2">&quot;html&quot;</span><span class="p">)</span>
<span class="gp">&gt;&gt;&gt; </span><span class="k">for</span> <span class="n">scope</span> <span class="ow">in</span> <span class="n">sel</span><span class="o">.</span><span class="n">xpath</span><span class="p">(</span><span class="s1">&#39;//div[@itemscope]&#39;</span><span class="p">):</span>
<span class="gp">... </span>    <span class="nb">print</span><span class="p">(</span><span class="s2">&quot;current scope:&quot;</span><span class="p">,</span> <span class="n">scope</span><span class="o">.</span><span class="n">xpath</span><span class="p">(</span><span class="s1">&#39;@itemtype&#39;</span><span class="p">)</span><span class="o">.</span><span class="n">getall</span><span class="p">())</span>
<span class="gp">... </span>    <span class="n">props</span> <span class="o">=</span> <span class="n">scope</span><span class="o">.</span><span class="n">xpath</span><span class="p">(</span><span class="s1">&#39;&#39;&#39;</span>
<span class="gp">... </span><span class="s1">                set:difference(./descendant::*/@itemprop,</span>
<span class="gp">... </span><span class="s1">                               .//*[@itemscope]/*/@itemprop)&#39;&#39;&#39;</span><span class="p">)</span>
<span class="gp">... </span>    <span class="nb">print</span><span class="p">(</span><span class="sa">f</span><span class="s2">&quot;    properties: </span><span class="si">{</span><span class="n">props</span><span class="o">.</span><span class="n">getall</span><span class="p">()</span><span class="si">}</span><span class="s2">&quot;</span><span class="p">)</span>
<span class="gp">... </span>    <span class="nb">print</span><span class="p">(</span><span class="s2">&quot;&quot;</span><span class="p">)</span>

<span class="go">current scope: [&#39;http://schema.org/Product&#39;]</span>
<span class="go">    properties: [&#39;name&#39;, &#39;aggregateRating&#39;, &#39;offers&#39;, &#39;description&#39;, &#39;review&#39;, &#39;review&#39;]</span>

<span class="go">current scope: [&#39;http://schema.org/AggregateRating&#39;]</span>
<span class="go">    properties: [&#39;ratingValue&#39;, &#39;reviewCount&#39;]</span>

<span class="go">current scope: [&#39;http://schema.org/Offer&#39;]</span>
<span class="go">    properties: [&#39;price&#39;, &#39;availability&#39;]</span>

<span class="go">current scope: [&#39;http://schema.org/Review&#39;]</span>
<span class="go">    properties: [&#39;name&#39;, &#39;author&#39;, &#39;datePublished&#39;, &#39;reviewRating&#39;, &#39;description&#39;]</span>

<span class="go">current scope: [&#39;http://schema.org/Rating&#39;]</span>
<span class="go">    properties: [&#39;worstRating&#39;, &#39;ratingValue&#39;, &#39;bestRating&#39;]</span>

<span class="go">current scope: [&#39;http://schema.org/Review&#39;]</span>
<span class="go">    properties: [&#39;name&#39;, &#39;author&#39;, &#39;datePublished&#39;, &#39;reviewRating&#39;, &#39;description&#39;]</span>

<span class="go">current scope: [&#39;http://schema.org/Rating&#39;]</span>
<span class="go">    properties: [&#39;worstRating&#39;, &#39;ratingValue&#39;, &#39;bestRating&#39;]</span>
</pre></div>
</div>
<p>在这里我们首先迭代 <code class="docutils literal notranslate"><span class="pre">itemscope</span></code> 元素，每一个元素，我们都在寻找 <code class="docutils literal notranslate"><span class="pre">itemprops</span></code> 元素并排除它们本身在另一个元素中的元素 <code class="docutils literal notranslate"><span class="pre">itemscope</span></code> .</p>
</div>
</div>
<div class="section" id="other-xpath-extensions">
<h3>其他XPath扩展<a class="headerlink" href="#other-xpath-extensions" title="永久链接至标题">¶</a></h3>
<p>scrapy选择器还提供一个非常遗漏的xpath扩展函数 <code class="docutils literal notranslate"><span class="pre">has-class</span></code> 它会回来 <code class="docutils literal notranslate"><span class="pre">True</span></code> 对于具有所有指定HTML类的节点。</p>
<p>对于以下HTML:：</p>
<div class="highlight-html notranslate"><div class="highlight"><pre><span></span><span class="p">&lt;</span><span class="nt">p</span> <span class="na">class</span><span class="o">=</span><span class="s">&quot;foo bar-baz&quot;</span><span class="p">&gt;</span>First<span class="p">&lt;/</span><span class="nt">p</span><span class="p">&gt;</span>
<span class="p">&lt;</span><span class="nt">p</span> <span class="na">class</span><span class="o">=</span><span class="s">&quot;foo&quot;</span><span class="p">&gt;</span>Second<span class="p">&lt;/</span><span class="nt">p</span><span class="p">&gt;</span>
<span class="p">&lt;</span><span class="nt">p</span> <span class="na">class</span><span class="o">=</span><span class="s">&quot;bar&quot;</span><span class="p">&gt;</span>Third<span class="p">&lt;/</span><span class="nt">p</span><span class="p">&gt;</span>
<span class="p">&lt;</span><span class="nt">p</span><span class="p">&gt;</span>Fourth<span class="p">&lt;/</span><span class="nt">p</span><span class="p">&gt;</span>
</pre></div>
</div>
<p>你可以这样使用它：</p>
<div class="doctest highlight-default notranslate"><div class="highlight"><pre><span></span><span class="gp">&gt;&gt;&gt; </span><span class="n">response</span><span class="o">.</span><span class="n">xpath</span><span class="p">(</span><span class="s1">&#39;//p[has-class(&quot;foo&quot;)]&#39;</span><span class="p">)</span>
<span class="go">[&lt;Selector xpath=&#39;//p[has-class(&quot;foo&quot;)]&#39; data=&#39;&lt;p class=&quot;foo bar-baz&quot;&gt;First&lt;/p&gt;&#39;&gt;,</span>
<span class="go"> &lt;Selector xpath=&#39;//p[has-class(&quot;foo&quot;)]&#39; data=&#39;&lt;p class=&quot;foo&quot;&gt;Second&lt;/p&gt;&#39;&gt;]</span>
<span class="gp">&gt;&gt;&gt; </span><span class="n">response</span><span class="o">.</span><span class="n">xpath</span><span class="p">(</span><span class="s1">&#39;//p[has-class(&quot;foo&quot;, &quot;bar-baz&quot;)]&#39;</span><span class="p">)</span>
<span class="go">[&lt;Selector xpath=&#39;//p[has-class(&quot;foo&quot;, &quot;bar-baz&quot;)]&#39; data=&#39;&lt;p class=&quot;foo bar-baz&quot;&gt;First&lt;/p&gt;&#39;&gt;]</span>
<span class="gp">&gt;&gt;&gt; </span><span class="n">response</span><span class="o">.</span><span class="n">xpath</span><span class="p">(</span><span class="s1">&#39;//p[has-class(&quot;foo&quot;, &quot;bar&quot;)]&#39;</span><span class="p">)</span>
<span class="go">[]</span>
</pre></div>
</div>
<p>所以XPath <code class="docutils literal notranslate"><span class="pre">//p[has-class(&quot;foo&quot;,</span> <span class="pre">&quot;bar-baz&quot;)]</span></code> 大致相当于CSS <code class="docutils literal notranslate"><span class="pre">p.foo.bar-baz</span></code> .  请注意，在大多数情况下，它的速度较慢，因为它是一个纯Python函数，可以为问题中的每个节点调用，而CSS查找被转换为xpath，因此运行效率更高，因此性能方面，它的使用仅限于不容易用css选择器描述的情况。</p>
<p>Parsel还简化了添加自己的xpath扩展。</p>
<dl class="py function">
<dt id="parsel.xpathfuncs.set_xpathfunc">
<code class="sig-prename descclassname">parsel.xpathfuncs.</code><code class="sig-name descname">set_xpathfunc</code><span class="sig-paren">(</span><em class="sig-param"><span class="n">fname</span></em>, <em class="sig-param"><span class="n">func</span></em><span class="sig-paren">)</span><a class="reference internal" href="../_modules/parsel/xpathfuncs.html#set_xpathfunc"><span class="viewcode-link">[源代码]</span></a><a class="headerlink" href="#parsel.xpathfuncs.set_xpathfunc" title="永久链接至目标">¶</a></dt>
<dd><p>注册要在xpath表达式中使用的自定义扩展函数。</p>
<p>函数 <code class="docutils literal notranslate"><span class="pre">func</span></code> 注册于 <code class="docutils literal notranslate"><span class="pre">fname</span></code> 将为每个匹配节点调用标识符，并将其传递给 <code class="docutils literal notranslate"><span class="pre">context</span></code> 参数以及从相应的xpath表达式传递的任何参数。</p>
<p>如果 <code class="docutils literal notranslate"><span class="pre">func</span></code> 是 <code class="docutils literal notranslate"><span class="pre">None</span></code> ，将删除扩展功能。</p>
<p>查看更多 <a class="reference external" href="http://lxml.de/extensions.html#xpath-extension-functions">in lxml documentation</a> .</p>
</dd></dl>

</div>
</div>
<div class="section" id="module-scrapy.selector">
<span id="built-in-selectors-reference"></span><span id="topics-selectors-ref"></span><h2>内置选择器引用<a class="headerlink" href="#module-scrapy.selector" title="永久链接至标题">¶</a></h2>
<div class="section" id="selector-objects">
<h3>选择器对象<a class="headerlink" href="#selector-objects" title="永久链接至标题">¶</a></h3>
<dl class="py class">
<dt id="scrapy.selector.Selector">
<em class="property">class </em><code class="sig-prename descclassname">scrapy.selector.</code><code class="sig-name descname">Selector</code><span class="sig-paren">(</span><em class="sig-param"><span class="o">*</span><span class="n">args</span></em>, <em class="sig-param"><span class="o">**</span><span class="n">kwargs</span></em><span class="sig-paren">)</span><a class="reference internal" href="../_modules/scrapy/selector/unified.html#Selector"><span class="viewcode-link">[源代码]</span></a><a class="headerlink" href="#scrapy.selector.Selector" title="永久链接至目标">¶</a></dt>
<dd><p>的实例 <a class="reference internal" href="#scrapy.selector.Selector" title="scrapy.selector.Selector"><code class="xref py py-class docutils literal notranslate"><span class="pre">Selector</span></code></a> 是一个包装响应，用于选择其内容的某些部分。</p>
<p><code class="docutils literal notranslate"><span class="pre">response</span></code> 是一个 <a class="reference internal" href="request-response.html#scrapy.http.HtmlResponse" title="scrapy.http.HtmlResponse"><code class="xref py py-class docutils literal notranslate"><span class="pre">HtmlResponse</span></code></a> 或 <a class="reference internal" href="request-response.html#scrapy.http.XmlResponse" title="scrapy.http.XmlResponse"><code class="xref py py-class docutils literal notranslate"><span class="pre">XmlResponse</span></code></a> 将用于选择和提取数据的对象。</p>
<p><code class="docutils literal notranslate"><span class="pre">text</span></code> 是Unicode字符串或UTF-8编码文本，用于 <code class="docutils literal notranslate"><span class="pre">response</span></code> 不可用。使用 <code class="docutils literal notranslate"><span class="pre">text</span></code> 和 <code class="docutils literal notranslate"><span class="pre">response</span></code> 一起是未定义的行为。</p>
<p><code class="docutils literal notranslate"><span class="pre">type</span></code> 定义选择器类型，它可以是 <code class="docutils literal notranslate"><span class="pre">&quot;html&quot;</span></code> ， <code class="docutils literal notranslate"><span class="pre">&quot;xml&quot;</span></code> 或 <code class="docutils literal notranslate"><span class="pre">None</span></code> （默认）。</p>
<p>如果 <code class="docutils literal notranslate"><span class="pre">type</span></code> 是 <code class="docutils literal notranslate"><span class="pre">None</span></code> ，选择器根据 <code class="docutils literal notranslate"><span class="pre">response</span></code> 类型（见下文），或默认为 <code class="docutils literal notranslate"><span class="pre">&quot;html&quot;</span></code> 以防与 <code class="docutils literal notranslate"><span class="pre">text</span></code> .</p>
<p>如果 <code class="docutils literal notranslate"><span class="pre">type</span></code> 是 <code class="docutils literal notranslate"><span class="pre">None</span></code> 和A <code class="docutils literal notranslate"><span class="pre">response</span></code> 如果传递，则从响应类型推断选择器类型，如下所示：</p>
<ul class="simple">
<li><p><code class="docutils literal notranslate"><span class="pre">&quot;html&quot;</span></code> 对于 <a class="reference internal" href="request-response.html#scrapy.http.HtmlResponse" title="scrapy.http.HtmlResponse"><code class="xref py py-class docutils literal notranslate"><span class="pre">HtmlResponse</span></code></a> 类型</p></li>
<li><p><code class="docutils literal notranslate"><span class="pre">&quot;xml&quot;</span></code> 对于 <a class="reference internal" href="request-response.html#scrapy.http.XmlResponse" title="scrapy.http.XmlResponse"><code class="xref py py-class docutils literal notranslate"><span class="pre">XmlResponse</span></code></a> 类型</p></li>
<li><p><code class="docutils literal notranslate"><span class="pre">&quot;html&quot;</span></code> 还有什么事吗</p></li>
</ul>
<p>否则，如果 <code class="docutils literal notranslate"><span class="pre">type</span></code> 设置后，选择器类型将被强制，不会发生检测。</p>
<dl class="py method">
<dt id="scrapy.selector.Selector.xpath">
<code class="sig-name descname">xpath</code><span class="sig-paren">(</span><em class="sig-param"><span class="n">query</span></em>, <em class="sig-param"><span class="n">namespaces</span><span class="o">=</span><span class="default_value">None</span></em>, <em class="sig-param"><span class="o">**</span><span class="n">kwargs</span></em><span class="sig-paren">)</span><a class="reference internal" href="../_modules/parsel/selector.html#Selector.xpath"><span class="viewcode-link">[源代码]</span></a><a class="headerlink" href="#scrapy.selector.Selector.xpath" title="永久链接至目标">¶</a></dt>
<dd><p>查找与xpath匹配的节点 <code class="docutils literal notranslate"><span class="pre">query</span></code> 并将结果作为 <a class="reference internal" href="#scrapy.selector.SelectorList" title="scrapy.selector.SelectorList"><code class="xref py py-class docutils literal notranslate"><span class="pre">SelectorList</span></code></a> 将所有元素展平的实例。列表元素实现 <a class="reference internal" href="#scrapy.selector.Selector" title="scrapy.selector.Selector"><code class="xref py py-class docutils literal notranslate"><span class="pre">Selector</span></code></a> 接口也是如此。</p>
<p><code class="docutils literal notranslate"><span class="pre">query</span></code> 是包含要应用的XPath查询的字符串。</p>
<p><code class="docutils literal notranslate"><span class="pre">namespaces</span></code> 是可选的 <code class="docutils literal notranslate"><span class="pre">prefix:</span> <span class="pre">namespace-uri</span></code> 将附加前缀的映射（dict）映射到 <code class="docutils literal notranslate"><span class="pre">register_namespace(prefix,</span> <span class="pre">uri)</span></code> . 相反 <code class="docutils literal notranslate"><span class="pre">register_namespace()</span></code> ，这些前缀不会保存以备将来调用。</p>
<p>可以使用任何其他命名参数来传递xpath表达式中xpath变量的值，例如：：</p>
<div class="highlight-python notranslate"><div class="highlight"><pre><span></span><span class="n">selector</span><span class="o">.</span><span class="n">xpath</span><span class="p">(</span><span class="s1">&#39;//a[href=$url]&#39;</span><span class="p">,</span> <span class="n">url</span><span class="o">=</span><span class="s2">&quot;http://www.example.com&quot;</span><span class="p">)</span>
</pre></div>
</div>
<div class="admonition note">
<p class="admonition-title">注解</p>
<p>为了方便起见，此方法可以调用为 <code class="docutils literal notranslate"><span class="pre">response.xpath()</span></code></p>
</div>
</dd></dl>

<dl class="py method">
<dt id="scrapy.selector.Selector.css">
<code class="sig-name descname">css</code><span class="sig-paren">(</span><em class="sig-param"><span class="n">query</span></em><span class="sig-paren">)</span><a class="reference internal" href="../_modules/parsel/selector.html#Selector.css"><span class="viewcode-link">[源代码]</span></a><a class="headerlink" href="#scrapy.selector.Selector.css" title="永久链接至目标">¶</a></dt>
<dd><p>应用给定的css选择器并返回 <a class="reference internal" href="#scrapy.selector.SelectorList" title="scrapy.selector.SelectorList"><code class="xref py py-class docutils literal notranslate"><span class="pre">SelectorList</span></code></a> 实例。</p>
<p><code class="docutils literal notranslate"><span class="pre">query</span></code> 是包含要应用的CSS选择器的字符串。</p>
<p>在后台，使用 <a class="reference external" href="https://pypi.python.org/pypi/cssselect/">cssselect</a> 类库与运行 <code class="docutils literal notranslate"><span class="pre">.xpath()</span></code> 方法。</p>
<div class="admonition note">
<p class="admonition-title">注解</p>
<p>为了方便起见，此方法可以调用为 <code class="docutils literal notranslate"><span class="pre">response.css()</span></code></p>
</div>
</dd></dl>

<dl class="py method">
<dt id="scrapy.selector.Selector.get">
<code class="sig-name descname">get</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="reference internal" href="../_modules/parsel/selector.html#Selector.get"><span class="viewcode-link">[源代码]</span></a><a class="headerlink" href="#scrapy.selector.Selector.get" title="永久链接至目标">¶</a></dt>
<dd><p>序列化并返回单个Unicode字符串中匹配的节点。未引用编码内容的百分比。</p>
<p>参见： <a class="reference internal" href="#old-extraction-api"><span class="std std-ref">extract（）和extract_first（）。</span></a></p>
</dd></dl>

<dl class="py attribute">
<dt id="scrapy.selector.Selector.attrib">
<code class="sig-name descname">attrib</code><a class="headerlink" href="#scrapy.selector.Selector.attrib" title="永久链接至目标">¶</a></dt>
<dd><p>返回基础元素的属性字典。</p>
<p>参见： <a class="reference internal" href="#selecting-attributes"><span class="std std-ref">选择元素属性</span></a> .</p>
</dd></dl>

<dl class="py method">
<dt id="scrapy.selector.Selector.re">
<code class="sig-name descname">re</code><span class="sig-paren">(</span><em class="sig-param"><span class="n">regex</span></em>, <em class="sig-param"><span class="n">replace_entities</span><span class="o">=</span><span class="default_value">True</span></em><span class="sig-paren">)</span><a class="reference internal" href="../_modules/parsel/selector.html#Selector.re"><span class="viewcode-link">[源代码]</span></a><a class="headerlink" href="#scrapy.selector.Selector.re" title="永久链接至目标">¶</a></dt>
<dd><p>应用给定的regex并返回带有匹配项的Unicode字符串列表。</p>
<p><code class="docutils literal notranslate"><span class="pre">regex</span></code> 可以是已编译的正则表达式，也可以是将使用 <code class="docutils literal notranslate"><span class="pre">re.compile(regex)</span></code> .</p>
<p>默认情况下，字符实体引用替换为其相应的字符（除了 <code class="docutils literal notranslate"><span class="pre">&amp;amp;</span></code> 和 <code class="docutils literal notranslate"><span class="pre">&amp;lt;</span></code> ）经过 <code class="docutils literal notranslate"><span class="pre">replace_entities</span></code> 作为 <code class="docutils literal notranslate"><span class="pre">False</span></code> 关闭这些替换。</p>
</dd></dl>

<dl class="py method">
<dt id="scrapy.selector.Selector.re_first">
<code class="sig-name descname">re_first</code><span class="sig-paren">(</span><em class="sig-param"><span class="n">regex</span></em>, <em class="sig-param"><span class="n">default</span><span class="o">=</span><span class="default_value">None</span></em>, <em class="sig-param"><span class="n">replace_entities</span><span class="o">=</span><span class="default_value">True</span></em><span class="sig-paren">)</span><a class="reference internal" href="../_modules/parsel/selector.html#Selector.re_first"><span class="viewcode-link">[源代码]</span></a><a class="headerlink" href="#scrapy.selector.Selector.re_first" title="永久链接至目标">¶</a></dt>
<dd><p>应用给定的regex并返回第一个匹配的unicode字符串。如果没有匹配项，则返回默认值 (<code class="docutils literal notranslate"><span class="pre">None</span></code> 如果未提供参数）。</p>
<p>默认情况下，字符实体引用替换为其相应的字符（除了 <code class="docutils literal notranslate"><span class="pre">&amp;amp;</span></code> 和 <code class="docutils literal notranslate"><span class="pre">&amp;lt;</span></code> ）经过 <code class="docutils literal notranslate"><span class="pre">replace_entities</span></code> 作为 <code class="docutils literal notranslate"><span class="pre">False</span></code> 关闭这些替换。</p>
</dd></dl>

<dl class="py method">
<dt id="scrapy.selector.Selector.register_namespace">
<code class="sig-name descname">register_namespace</code><span class="sig-paren">(</span><em class="sig-param"><span class="n">prefix</span></em>, <em class="sig-param"><span class="n">uri</span></em><span class="sig-paren">)</span><a class="reference internal" href="../_modules/parsel/selector.html#Selector.register_namespace"><span class="viewcode-link">[源代码]</span></a><a class="headerlink" href="#scrapy.selector.Selector.register_namespace" title="永久链接至目标">¶</a></dt>
<dd><p>注册要在此中使用的给定命名空间 <a class="reference internal" href="#scrapy.selector.Selector" title="scrapy.selector.Selector"><code class="xref py py-class docutils literal notranslate"><span class="pre">Selector</span></code></a> . 如果不注册命名空间，则无法从非标准命名空间中选择或提取数据。见 <a class="reference internal" href="#selector-examples-xml"><span class="std std-ref">XML响应的选择器示例</span></a> .</p>
</dd></dl>

<dl class="py method">
<dt id="scrapy.selector.Selector.remove_namespaces">
<code class="sig-name descname">remove_namespaces</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="reference internal" href="../_modules/parsel/selector.html#Selector.remove_namespaces"><span class="viewcode-link">[源代码]</span></a><a class="headerlink" href="#scrapy.selector.Selector.remove_namespaces" title="永久链接至目标">¶</a></dt>
<dd><p>删除所有名称空间，允许使用不含名称空间的xpaths遍历文档。见 <a class="reference internal" href="#removing-namespaces"><span class="std std-ref">正在删除命名空间</span></a> .</p>
</dd></dl>

<dl class="py method">
<dt id="scrapy.selector.Selector.__bool__">
<code class="sig-name descname">__bool__</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="reference internal" href="../_modules/parsel/selector.html#Selector.__bool__"><span class="viewcode-link">[源代码]</span></a><a class="headerlink" href="#scrapy.selector.Selector.__bool__" title="永久链接至目标">¶</a></dt>
<dd><p>返回 <code class="docutils literal notranslate"><span class="pre">True</span></code> 如果选择了任何真实内容或 <code class="docutils literal notranslate"><span class="pre">False</span></code> 否则。换句话说，布尔值 <a class="reference internal" href="#scrapy.selector.Selector" title="scrapy.selector.Selector"><code class="xref py py-class docutils literal notranslate"><span class="pre">Selector</span></code></a> 由它选择的内容给出。</p>
</dd></dl>

<dl class="py method">
<dt id="scrapy.selector.Selector.getall">
<code class="sig-name descname">getall</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="reference internal" href="../_modules/parsel/selector.html#Selector.getall"><span class="viewcode-link">[源代码]</span></a><a class="headerlink" href="#scrapy.selector.Selector.getall" title="永久链接至目标">¶</a></dt>
<dd><p>序列化并返回unicode字符串的1元素列表中匹配的节点。</p>
<p>为了保持一致性，这个方法被添加到选择器中；它对于选择器列表更有用。参见： <a class="reference internal" href="#old-extraction-api"><span class="std std-ref">extract（）和extract_first（）。</span></a></p>
</dd></dl>

</dd></dl>

</div>
<div class="section" id="selectorlist-objects">
<h3>SelectorList对象<a class="headerlink" href="#selectorlist-objects" title="永久链接至标题">¶</a></h3>
<dl class="py class">
<dt id="scrapy.selector.SelectorList">
<em class="property">class </em><code class="sig-prename descclassname">scrapy.selector.</code><code class="sig-name descname">SelectorList</code><span class="sig-paren">(</span><em class="sig-param">iterable=()</em>, <em class="sig-param">/</em><span class="sig-paren">)</span><a class="reference internal" href="../_modules/scrapy/selector/unified.html#SelectorList"><span class="viewcode-link">[源代码]</span></a><a class="headerlink" href="#scrapy.selector.SelectorList" title="永久链接至目标">¶</a></dt>
<dd><p>这个 <a class="reference internal" href="#scrapy.selector.SelectorList" title="scrapy.selector.SelectorList"><code class="xref py py-class docutils literal notranslate"><span class="pre">SelectorList</span></code></a> 类是内置的子类 <code class="docutils literal notranslate"><span class="pre">list</span></code> 类，它提供了一些附加方法。</p>
<dl class="py method">
<dt id="scrapy.selector.SelectorList.xpath">
<code class="sig-name descname">xpath</code><span class="sig-paren">(</span><em class="sig-param"><span class="n">xpath</span></em>, <em class="sig-param"><span class="n">namespaces</span><span class="o">=</span><span class="default_value">None</span></em>, <em class="sig-param"><span class="o">**</span><span class="n">kwargs</span></em><span class="sig-paren">)</span><a class="reference internal" href="../_modules/parsel/selector.html#SelectorList.xpath"><span class="viewcode-link">[源代码]</span></a><a class="headerlink" href="#scrapy.selector.SelectorList.xpath" title="永久链接至目标">¶</a></dt>
<dd><p>打电话给 <code class="docutils literal notranslate"><span class="pre">.xpath()</span></code> 此列表中的每个元素的方法，并将其结果扁平化为另一个 <a class="reference internal" href="#scrapy.selector.SelectorList" title="scrapy.selector.SelectorList"><code class="xref py py-class docutils literal notranslate"><span class="pre">SelectorList</span></code></a> .</p>
<p><code class="docutils literal notranslate"><span class="pre">query</span></code> is the same argument as the one in <a class="reference internal" href="#scrapy.selector.Selector.xpath" title="scrapy.selector.Selector.xpath"><code class="xref py py-meth docutils literal notranslate"><span class="pre">Selector.xpath()</span></code></a></p>
<p><code class="docutils literal notranslate"><span class="pre">namespaces</span></code> 是可选的 <code class="docutils literal notranslate"><span class="pre">prefix:</span> <span class="pre">namespace-uri</span></code> 将附加前缀的映射（dict）映射到 <code class="docutils literal notranslate"><span class="pre">register_namespace(prefix,</span> <span class="pre">uri)</span></code> . 相反 <code class="docutils literal notranslate"><span class="pre">register_namespace()</span></code> ，这些前缀不会保存以备将来调用。</p>
<p>可以使用任何其他命名参数来传递xpath表达式中xpath变量的值，例如：：</p>
<div class="highlight-python notranslate"><div class="highlight"><pre><span></span><span class="n">selector</span><span class="o">.</span><span class="n">xpath</span><span class="p">(</span><span class="s1">&#39;//a[href=$url]&#39;</span><span class="p">,</span> <span class="n">url</span><span class="o">=</span><span class="s2">&quot;http://www.example.com&quot;</span><span class="p">)</span>
</pre></div>
</div>
</dd></dl>

<dl class="py method">
<dt id="scrapy.selector.SelectorList.css">
<code class="sig-name descname">css</code><span class="sig-paren">(</span><em class="sig-param"><span class="n">query</span></em><span class="sig-paren">)</span><a class="reference internal" href="../_modules/parsel/selector.html#SelectorList.css"><span class="viewcode-link">[源代码]</span></a><a class="headerlink" href="#scrapy.selector.SelectorList.css" title="永久链接至目标">¶</a></dt>
<dd><p>打电话给 <code class="docutils literal notranslate"><span class="pre">.css()</span></code> 此列表中的每个元素的方法，并将其结果扁平化为另一个 <a class="reference internal" href="#scrapy.selector.SelectorList" title="scrapy.selector.SelectorList"><code class="xref py py-class docutils literal notranslate"><span class="pre">SelectorList</span></code></a> .</p>
<p><code class="docutils literal notranslate"><span class="pre">query</span></code> is the same argument as the one in <a class="reference internal" href="#scrapy.selector.Selector.css" title="scrapy.selector.Selector.css"><code class="xref py py-meth docutils literal notranslate"><span class="pre">Selector.css()</span></code></a></p>
</dd></dl>

<dl class="py method">
<dt id="scrapy.selector.SelectorList.getall">
<code class="sig-name descname">getall</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="reference internal" href="../_modules/parsel/selector.html#SelectorList.getall"><span class="viewcode-link">[源代码]</span></a><a class="headerlink" href="#scrapy.selector.SelectorList.getall" title="永久链接至目标">¶</a></dt>
<dd><p>打电话给 <code class="docutils literal notranslate"><span class="pre">.get()</span></code> 每个元素的方法都是这个列表，并将它们的结果作为一个Unicode字符串列表平展地返回。</p>
<p>参见： <a class="reference internal" href="#old-extraction-api"><span class="std std-ref">extract（）和extract_first（）。</span></a></p>
</dd></dl>

<dl class="py method">
<dt id="scrapy.selector.SelectorList.get">
<code class="sig-name descname">get</code><span class="sig-paren">(</span><em class="sig-param"><span class="n">default</span><span class="o">=</span><span class="default_value">None</span></em><span class="sig-paren">)</span><a class="reference internal" href="../_modules/parsel/selector.html#SelectorList.get"><span class="viewcode-link">[源代码]</span></a><a class="headerlink" href="#scrapy.selector.SelectorList.get" title="永久链接至目标">¶</a></dt>
<dd><p>返回的结果 <code class="docutils literal notranslate"><span class="pre">.get()</span></code> 对于此列表中的第一个元素。如果列表为空，则返回默认值。</p>
<p>参见： <a class="reference internal" href="#old-extraction-api"><span class="std std-ref">extract（）和extract_first（）。</span></a></p>
</dd></dl>

<dl class="py method">
<dt id="scrapy.selector.SelectorList.re">
<code class="sig-name descname">re</code><span class="sig-paren">(</span><em class="sig-param"><span class="n">regex</span></em>, <em class="sig-param"><span class="n">replace_entities</span><span class="o">=</span><span class="default_value">True</span></em><span class="sig-paren">)</span><a class="reference internal" href="../_modules/parsel/selector.html#SelectorList.re"><span class="viewcode-link">[源代码]</span></a><a class="headerlink" href="#scrapy.selector.SelectorList.re" title="永久链接至目标">¶</a></dt>
<dd><p>打电话给 <code class="docutils literal notranslate"><span class="pre">.re()</span></code> 方法，并以unicode字符串列表的形式返回结果。</p>
<p>默认情况下，字符实体引用替换为其相应的字符（除了 <code class="docutils literal notranslate"><span class="pre">&amp;amp;</span></code> 和 <code class="docutils literal notranslate"><span class="pre">&amp;lt;</span></code> .经过 <code class="docutils literal notranslate"><span class="pre">replace_entities</span></code> 作为 <code class="docutils literal notranslate"><span class="pre">False</span></code> 关闭这些替换。</p>
</dd></dl>

<dl class="py method">
<dt id="scrapy.selector.SelectorList.re_first">
<code class="sig-name descname">re_first</code><span class="sig-paren">(</span><em class="sig-param"><span class="n">regex</span></em>, <em class="sig-param"><span class="n">default</span><span class="o">=</span><span class="default_value">None</span></em>, <em class="sig-param"><span class="n">replace_entities</span><span class="o">=</span><span class="default_value">True</span></em><span class="sig-paren">)</span><a class="reference internal" href="../_modules/parsel/selector.html#SelectorList.re_first"><span class="viewcode-link">[源代码]</span></a><a class="headerlink" href="#scrapy.selector.SelectorList.re_first" title="永久链接至目标">¶</a></dt>
<dd><p>打电话给 <code class="docutils literal notranslate"><span class="pre">.re()</span></code> 方法，并以Unicode字符串返回结果。如果列表为空或regex不匹配，则返回默认值 (<code class="docutils literal notranslate"><span class="pre">None</span></code> 如果未提供参数）。</p>
<p>默认情况下，字符实体引用替换为其相应的字符（除了 <code class="docutils literal notranslate"><span class="pre">&amp;amp;</span></code> 和 <code class="docutils literal notranslate"><span class="pre">&amp;lt;</span></code> .经过 <code class="docutils literal notranslate"><span class="pre">replace_entities</span></code> 作为 <code class="docutils literal notranslate"><span class="pre">False</span></code> 关闭这些替换。</p>
</dd></dl>

<dl class="py attribute">
<dt id="scrapy.selector.SelectorList.attrib">
<code class="sig-name descname">attrib</code><a class="headerlink" href="#scrapy.selector.SelectorList.attrib" title="永久链接至目标">¶</a></dt>
<dd><p>返回第一个元素的属性字典。如果列表为空，则返回空的dict。</p>
<p>参见： <a class="reference internal" href="#selecting-attributes"><span class="std std-ref">选择元素属性</span></a> .</p>
</dd></dl>

</dd></dl>

</div>
</div>
<div class="section" id="examples">
<span id="selector-examples"></span><h2>实例<a class="headerlink" href="#examples" title="永久链接至标题">¶</a></h2>
<div class="section" id="selector-examples-on-html-response">
<span id="selector-examples-html"></span><h3>HTML响应的选择器示例<a class="headerlink" href="#selector-examples-on-html-response" title="永久链接至标题">¶</a></h3>
<p>这里有一些 <a class="reference internal" href="#scrapy.selector.Selector" title="scrapy.selector.Selector"><code class="xref py py-class docutils literal notranslate"><span class="pre">Selector</span></code></a> 举例说明几个概念。在所有情况下，我们假设 <a class="reference internal" href="#scrapy.selector.Selector" title="scrapy.selector.Selector"><code class="xref py py-class docutils literal notranslate"><span class="pre">Selector</span></code></a> 用一个 <a class="reference internal" href="request-response.html#scrapy.http.HtmlResponse" title="scrapy.http.HtmlResponse"><code class="xref py py-class docutils literal notranslate"><span class="pre">HtmlResponse</span></code></a> 这样的对象：</p>
<div class="highlight-python notranslate"><div class="highlight"><pre><span></span><span class="n">sel</span> <span class="o">=</span> <span class="n">Selector</span><span class="p">(</span><span class="n">html_response</span><span class="p">)</span>
</pre></div>
</div>
<ol class="arabic">
<li><p>选择全部 <code class="docutils literal notranslate"><span class="pre">&lt;h1&gt;</span></code> 来自HTML响应正文的元素，返回 <a class="reference internal" href="#scrapy.selector.Selector" title="scrapy.selector.Selector"><code class="xref py py-class docutils literal notranslate"><span class="pre">Selector</span></code></a> 对象（即 <a class="reference internal" href="#scrapy.selector.SelectorList" title="scrapy.selector.SelectorList"><code class="xref py py-class docutils literal notranslate"><span class="pre">SelectorList</span></code></a> 对象）：</p>
<div class="highlight-python notranslate"><div class="highlight"><pre><span></span><span class="n">sel</span><span class="o">.</span><span class="n">xpath</span><span class="p">(</span><span class="s2">&quot;//h1&quot;</span><span class="p">)</span>
</pre></div>
</div>
</li>
<li><p>提取所有文本 <code class="docutils literal notranslate"><span class="pre">&lt;h1&gt;</span></code> 元素，返回字符串列表：</p>
<div class="highlight-python notranslate"><div class="highlight"><pre><span></span><span class="n">sel</span><span class="o">.</span><span class="n">xpath</span><span class="p">(</span><span class="s2">&quot;//h1&quot;</span><span class="p">)</span><span class="o">.</span><span class="n">getall</span><span class="p">()</span>         <span class="c1"># this includes the h1 tag</span>
<span class="n">sel</span><span class="o">.</span><span class="n">xpath</span><span class="p">(</span><span class="s2">&quot;//h1/text()&quot;</span><span class="p">)</span><span class="o">.</span><span class="n">getall</span><span class="p">()</span>  <span class="c1"># this excludes the h1 tag</span>
</pre></div>
</div>
</li>
<li><p>全部迭代 <code class="docutils literal notranslate"><span class="pre">&lt;p&gt;</span></code> 标记并打印其类属性：</p>
<div class="highlight-python notranslate"><div class="highlight"><pre><span></span><span class="k">for</span> <span class="n">node</span> <span class="ow">in</span> <span class="n">sel</span><span class="o">.</span><span class="n">xpath</span><span class="p">(</span><span class="s2">&quot;//p&quot;</span><span class="p">):</span>
    <span class="nb">print</span><span class="p">(</span><span class="n">node</span><span class="o">.</span><span class="n">attrib</span><span class="p">[</span><span class="s1">&#39;class&#39;</span><span class="p">])</span>
</pre></div>
</div>
</li>
</ol>
</div>
<div class="section" id="selector-examples-on-xml-response">
<span id="selector-examples-xml"></span><h3>XML响应的选择器示例<a class="headerlink" href="#selector-examples-on-xml-response" title="永久链接至标题">¶</a></h3>
<p>下面是一些例子来说明 <a class="reference internal" href="#scrapy.selector.Selector" title="scrapy.selector.Selector"><code class="xref py py-class docutils literal notranslate"><span class="pre">Selector</span></code></a> 对象用 <a class="reference internal" href="request-response.html#scrapy.http.XmlResponse" title="scrapy.http.XmlResponse"><code class="xref py py-class docutils literal notranslate"><span class="pre">XmlResponse</span></code></a> 对象：</p>
<div class="highlight-python notranslate"><div class="highlight"><pre><span></span><span class="n">sel</span> <span class="o">=</span> <span class="n">Selector</span><span class="p">(</span><span class="n">xml_response</span><span class="p">)</span>
</pre></div>
</div>
<ol class="arabic">
<li><p>选择全部 <code class="docutils literal notranslate"><span class="pre">&lt;product&gt;</span></code> 来自XML响应主体的元素，返回 <a class="reference internal" href="#scrapy.selector.Selector" title="scrapy.selector.Selector"><code class="xref py py-class docutils literal notranslate"><span class="pre">Selector</span></code></a> 对象（即 <a class="reference internal" href="#scrapy.selector.SelectorList" title="scrapy.selector.SelectorList"><code class="xref py py-class docutils literal notranslate"><span class="pre">SelectorList</span></code></a> 对象）：</p>
<div class="highlight-python notranslate"><div class="highlight"><pre><span></span><span class="n">sel</span><span class="o">.</span><span class="n">xpath</span><span class="p">(</span><span class="s2">&quot;//product&quot;</span><span class="p">)</span>
</pre></div>
</div>
</li>
<li><p>从A中提取所有价格 <a class="reference external" href="https://support.google.com/merchants/answer/160589?hl=en&amp;ref_topic=2473799">Google Base XML feed</a> 需要注册命名空间：：</p>
<div class="highlight-python notranslate"><div class="highlight"><pre><span></span><span class="n">sel</span><span class="o">.</span><span class="n">register_namespace</span><span class="p">(</span><span class="s2">&quot;g&quot;</span><span class="p">,</span> <span class="s2">&quot;http://base.google.com/ns/1.0&quot;</span><span class="p">)</span>
<span class="n">sel</span><span class="o">.</span><span class="n">xpath</span><span class="p">(</span><span class="s2">&quot;//g:price&quot;</span><span class="p">)</span><span class="o">.</span><span class="n">getall</span><span class="p">()</span>
</pre></div>
</div>
</li>
</ol>
</div>
</div>
</div>


           </div>
           
          </div>
          <footer>
  
    <div class="rst-footer-buttons" role="navigation" aria-label="footer navigation">
      
        <a href="items.html" class="btn btn-neutral float-right" title="项目" accesskey="n" rel="next">Next <span class="fa fa-arrow-circle-right"></span></a>
      
      
        <a href="spiders.html" class="btn btn-neutral float-left" title="蜘蛛" accesskey="p" rel="prev"><span class="fa fa-arrow-circle-left"></span> Previous</a>
      
    </div>
  

  <hr/>

  <div role="contentinfo">
    <p>
        
        &copy; 版权所有 2008–2020, Scrapy developers
      <span class="lastupdated">
        最后更新于 10月 18, 2020.
      </span>

    </p>
  </div>
    
    
    
    Built with <a href="http://sphinx-doc.org/">Sphinx</a> using a
    
    <a href="https://github.com/rtfd/sphinx_rtd_theme">theme</a>
    
    provided by <a href="https://readthedocs.org">Read the Docs</a>. 

</footer>

        </div>
      </div>

    </section>

  </div>
  

  <script type="text/javascript">
      jQuery(function () {
          SphinxRtdTheme.Navigation.enable(true);
      });
  </script>

  
  
    
  
 
<script type="text/javascript">
!function(){var analytics=window.analytics=window.analytics||[];if(!analytics.initialize)if(analytics.invoked)window.console&&console.error&&console.error("Segment snippet included twice.");else{analytics.invoked=!0;analytics.methods=["trackSubmit","trackClick","trackLink","trackForm","pageview","identify","reset","group","track","ready","alias","page","once","off","on"];analytics.factory=function(t){return function(){var e=Array.prototype.slice.call(arguments);e.unshift(t);analytics.push(e);return analytics}};for(var t=0;t<analytics.methods.length;t++){var e=analytics.methods[t];analytics[e]=analytics.factory(e)}analytics.load=function(t){var e=document.createElement("script");e.type="text/javascript";e.async=!0;e.src=("https:"===document.location.protocol?"https://":"http://")+"cdn.segment.com/analytics.js/v1/"+t+"/analytics.min.js";var n=document.getElementsByTagName("script")[0];n.parentNode.insertBefore(e,n)};analytics.SNIPPET_VERSION="3.1.0";
analytics.load("8UDQfnf3cyFSTsM4YANnW5sXmgZVILbA");
analytics.page();
}}();

analytics.ready(function () {
    ga('require', 'linker');
    ga('linker:autoLink', ['scrapinghub.com', 'crawlera.com']);
});
</script>


</body>
</html>